[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:24 dist-test-slave-9tkz krb5kdc[7954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:24 dist-test-slave-9tkz krb5kdc[7954](info): set up 2 sockets
Jun 19 02:34:24 dist-test-slave-9tkz krb5kdc[7954](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 02:34:25.338133 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:25 dist-test-slave-9tkz krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:25.481060 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44199
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:38939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:44199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:25.595793 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:25.596088 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:25.596129 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:25.600306 7967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:25.600380 7967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:25.600400 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:25.600417 7967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:25.600433 7967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:25.605813 7967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:38939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:44199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:25.606907 7967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:25.607928 7967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:25.616935 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:25.616904 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:25.616927 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:25.617162 7967 server_base.cc:1034] running on GCE node
I20260619 02:34:25.617839 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:25.619076 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:25.620270 7967 hybrid_clock.cc:648] HybridClock initialized: now 1781836465620218 us; error 68 us; skew 500 ppm
Jun 19 02:34:25 dist-test-slave-9tkz krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836465, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:25.623188 7967 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:25.624187 7967 webserver.cc:460] Webserver started at http://127.7.194.254:45599/ using document root <none> and password file <none>
I20260619 02:34:25.624778 7967 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:25.624873 7967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:25.625128 7967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:25.627444 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "a15feaf4721947558c73c2c68e9354b0"
format_stamp: "Formatted at 2026-06-19 02:34:25 on dist-test-slave-9tkz"
server_key: "4d519dfa72e1dc04e9bf39ffa5c2a487"
server_key_iv: "e7e457a5cbba5ba20a973025c894be94"
server_key_version: "encryptionkey@0"
I20260619 02:34:25.628052 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "a15feaf4721947558c73c2c68e9354b0"
format_stamp: "Formatted at 2026-06-19 02:34:25 on dist-test-slave-9tkz"
server_key: "4d519dfa72e1dc04e9bf39ffa5c2a487"
server_key_iv: "e7e457a5cbba5ba20a973025c894be94"
server_key_version: "encryptionkey@0"
I20260619 02:34:25.632154 7967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260619 02:34:25.634716 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:25.636070 7967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:25.636221 7967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "a15feaf4721947558c73c2c68e9354b0"
format_stamp: "Formatted at 2026-06-19 02:34:25 on dist-test-slave-9tkz"
server_key: "4d519dfa72e1dc04e9bf39ffa5c2a487"
server_key_iv: "e7e457a5cbba5ba20a973025c894be94"
server_key_version: "encryptionkey@0"
I20260619 02:34:25.636339 7967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:25.652014 7967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:25.655030 7967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:25.655251 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:25.662964 7967 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:44199
I20260619 02:34:25.662981 8034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:44199 every 8 connection(s)
I20260619 02:34:25.663970 7967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:25.666839 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:25.667991 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 7967
I20260619 02:34:25.668150 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:25.669461 7947 external_mini_cluster.cc:1428] Setting key 677bb7d058cbf62ec39513d58fe88ead
I20260619 02:34:25.673077 8035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0: Bootstrap starting.
I20260619 02:34:25.675573 8035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:25.676319 8035 log.cc:824] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:25.678258 8035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0: No bootstrap required, opened a new log
I20260619 02:34:25.681176 8035 raft_consensus.cc:348] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15feaf4721947558c73c2c68e9354b0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44199 } }
I20260619 02:34:25.681435 8035 raft_consensus.cc:374] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:25.681519 8035 raft_consensus.cc:729] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15feaf4721947558c73c2c68e9354b0, State: Initialized, Role: FOLLOWER
I20260619 02:34:25.681964 8035 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15feaf4721947558c73c2c68e9354b0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44199 } }
I20260619 02:34:25.682132 8035 raft_consensus.cc:388] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:25.682217 8035 raft_consensus.cc:482] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:25.682332 8035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:25.683207 8035 raft_consensus.cc:504] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15feaf4721947558c73c2c68e9354b0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44199 } }
I20260619 02:34:25.683554 8035 leader_election.cc:302] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a15feaf4721947558c73c2c68e9354b0; no voters:
I20260619 02:34:25.683853 8035 leader_election.cc:288] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:25.683995 8040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:25.684206 8040 raft_consensus.cc:686] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [term 1 LEADER]: Becoming Leader. State: Replica: a15feaf4721947558c73c2c68e9354b0, State: Running, Role: LEADER
I20260619 02:34:25.684536 8040 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15feaf4721947558c73c2c68e9354b0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44199 } }
I20260619 02:34:25.685103 8035 sys_catalog.cc:564] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:25.685786 8042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15feaf4721947558c73c2c68e9354b0. Latest consensus state: current_term: 1 leader_uuid: "a15feaf4721947558c73c2c68e9354b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15feaf4721947558c73c2c68e9354b0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44199 } } }
I20260619 02:34:25.685899 8042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:25.686246 8041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a15feaf4721947558c73c2c68e9354b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15feaf4721947558c73c2c68e9354b0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44199 } } }
I20260619 02:34:25.686322 8041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:25.687434 8046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 02:34:25 dist-test-slave-9tkz krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:25.691118 8046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:25.695436 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:25.672675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.7.194.254:44199)
0619 02:34:25.673106 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:25.673125 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:25.674235 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 02:34:25.677439 (+ 3204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:25.677455 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:25.677842 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:25.678186 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:25.680386 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:25.681913 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:25.683293 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:25.683898 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:25.684067 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:25.689412 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:25.689420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:25.689436 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:25.689465 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:25.691719 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:25.692341 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:25.692347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:25.692352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:25.692496 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:25.692951 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:25.692956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:25.692961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:25.693376 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:25.693633 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:25.694210 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 02:34:25.694490 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":158,"threads_started":1}
I20260619 02:34:25.698443 8046 catalog_manager.cc:1347] Generated new cluster ID: c46f47d37e2942af9af958853c11e132
I20260619 02:34:25.698529 8046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:25.712443 8046 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:25.713824 8046 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:25.720613 8046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a15feaf4721947558c73c2c68e9354b0: Generated new TSK 0
I20260619 02:34:25.721491 8046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:34:25.756440 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44199
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:38939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:25.870823 8061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:25.871158 8061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:25.871235 8061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:25.875432 8061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:25.875535 8061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:25.875634 8061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:25.881172 8061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:38939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:25.882424 8061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:25.883553 8061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:25.893754 8069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:25.893765 8067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:25.893743 8066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:25.894225 8061 server_base.cc:1034] running on GCE node
I20260619 02:34:25.894635 8061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:25.895251 8061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:25.896420 8061 hybrid_clock.cc:648] HybridClock initialized: now 1781836465896407 us; error 25 us; skew 500 ppm
Jun 19 02:34:25 dist-test-slave-9tkz krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836465, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:25.899379 8061 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:34:25.900202 8061 webserver.cc:460] Webserver started at http://127.7.194.193:44131/ using document root <none> and password file <none>
I20260619 02:34:25.900741 8061 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:25.900799 8061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:25.900966 8061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:25.903070 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "3e555c8c9fe3471788cbaeae75142199"
format_stamp: "Formatted at 2026-06-19 02:34:25 on dist-test-slave-9tkz"
server_key: "5e16aeb66c8ddd60dc898c7746c96ba7"
server_key_iv: "33f7044a1ab3f4375d292d12375f293c"
server_key_version: "encryptionkey@0"
I20260619 02:34:25.903604 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "3e555c8c9fe3471788cbaeae75142199"
format_stamp: "Formatted at 2026-06-19 02:34:25 on dist-test-slave-9tkz"
server_key: "5e16aeb66c8ddd60dc898c7746c96ba7"
server_key_iv: "33f7044a1ab3f4375d292d12375f293c"
server_key_version: "encryptionkey@0"
I20260619 02:34:25.907490 8061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:25.909921 8076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:25.911280 8061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:34:25.911427 8061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "3e555c8c9fe3471788cbaeae75142199"
format_stamp: "Formatted at 2026-06-19 02:34:25 on dist-test-slave-9tkz"
server_key: "5e16aeb66c8ddd60dc898c7746c96ba7"
server_key_iv: "33f7044a1ab3f4375d292d12375f293c"
server_key_version: "encryptionkey@0"
I20260619 02:34:25.911554 8061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:25.921471 8061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:25.924933 8061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:25.925148 8061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:25.925868 8061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:25.926724 8061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:25.926798 8061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:25.926867 8061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:25.926899 8061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:25.936369 8061 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:46593
I20260619 02:34:25.936424 8189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:46593 every 8 connection(s)
I20260619 02:34:25.937330 8061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:25.943122 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 8061
I20260619 02:34:25.943248 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:25.943517 7947 external_mini_cluster.cc:1428] Setting key 743c849c46a7f74af6a3a65d6ce3418d
Jun 19 02:34:25 dist-test-slave-9tkz krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836465, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:25.952723 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:25.939305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:46795 (local address 127.7.194.254:44199)
0619 02:34:25.939470 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:25.939473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:25.940361 (+ 888us) server_negotiation.cc:407] Connection header received
0619 02:34:25.941862 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:25.941866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:25.941927 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:25.942008 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:25.943226 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:25.943941 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:25.945475 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:25.945787 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:25.945872 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:25.948655 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:25.948662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:25.948668 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:25.948702 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:25.950559 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:25.951064 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:25.951070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:25.951075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:25.951172 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:25.951507 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:25.951512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:25.951516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:25.951760 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:25.951870 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:25.952368 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 02:34:25.952534 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:34:25.953758 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:25.939710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44199 (local address 127.7.194.193:46795)
0619 02:34:25.940199 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:25.940241 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:25.941632 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:25.942194 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:25.942201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:25.942569 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:25.943065 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:25.943079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:25.944086 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:25.944089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:25.945332 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:25.945344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:25.945944 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:25.945952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:25.946228 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:25.946988 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:25.947005 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:25.948477 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:25.950676 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:25.950680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:25.950694 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:25.950958 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:25.951266 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:25.951268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:25.951269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:25.951409 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:25.951862 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:25.951868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:25.952127 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 02:34:25.952280 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 02:34:25.952566 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":120,"threads_started":1}
I20260619 02:34:25.955207 8190 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44199
I20260619 02:34:25.955474 8190 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:25.956111 8190 heartbeater.cc:511] Master 127.7.194.254:44199 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:34:25.957988 7999 ts_manager.cc:194] Registered new tserver with Master: 3e555c8c9fe3471788cbaeae75142199 (127.7.194.193:46593)
I20260619 02:34:25.959177 7999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:46795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:34:25.974023 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44199
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:38939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:26.087819 8195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:26.088114 8195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:26.088160 8195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:26.092352 8195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:26.092435 8195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:26.092501 8195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:26.098106 8195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:38939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:26.099242 8195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:26.100389 8195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:26.110412 8203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:26.110409 8200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:26.110460 8201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:26.110803 8195 server_base.cc:1034] running on GCE node
I20260619 02:34:26.111225 8195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:26.111901 8195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:26.113092 8195 hybrid_clock.cc:648] HybridClock initialized: now 1781836466113082 us; error 36 us; skew 500 ppm
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836466, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:26.116096 8195 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:34:26.117057 8195 webserver.cc:460] Webserver started at http://127.7.194.194:41223/ using document root <none> and password file <none>
I20260619 02:34:26.117718 8195 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:26.117815 8195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:26.118054 8195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:26.120280 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "750abb2b4cc243018c04bdb18adb1bcb"
format_stamp: "Formatted at 2026-06-19 02:34:26 on dist-test-slave-9tkz"
server_key: "72f9bcc6575a79d0e9aede601c94f375"
server_key_iv: "26bf97da0df434d3a6cf99fcc39bdfe7"
server_key_version: "encryptionkey@0"
I20260619 02:34:26.120867 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "750abb2b4cc243018c04bdb18adb1bcb"
format_stamp: "Formatted at 2026-06-19 02:34:26 on dist-test-slave-9tkz"
server_key: "72f9bcc6575a79d0e9aede601c94f375"
server_key_iv: "26bf97da0df434d3a6cf99fcc39bdfe7"
server_key_version: "encryptionkey@0"
I20260619 02:34:26.124861 8195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:34:26.127523 8210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:26.128846 8195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:34:26.129001 8195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "750abb2b4cc243018c04bdb18adb1bcb"
format_stamp: "Formatted at 2026-06-19 02:34:26 on dist-test-slave-9tkz"
server_key: "72f9bcc6575a79d0e9aede601c94f375"
server_key_iv: "26bf97da0df434d3a6cf99fcc39bdfe7"
server_key_version: "encryptionkey@0"
I20260619 02:34:26.129123 8195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:26.142074 8195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:26.145218 8195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:26.145452 8195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:26.146099 8195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:26.146970 8195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:26.147044 8195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:26.147110 8195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:26.147145 8195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:26.156172 8195 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:36237
I20260619 02:34:26.156190 8323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:36237 every 8 connection(s)
I20260619 02:34:26.157124 8195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:26.160658 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 8195
I20260619 02:34:26.160801 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:26.161110 7947 external_mini_cluster.cc:1428] Setting key 58d396ec7d7053fac384f44a36bed95f
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836466, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:34:26.172753 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:26.159018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:45579 (local address 127.7.194.254:44199)
0619 02:34:26.159183 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:26.159187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:26.160030 (+ 843us) server_negotiation.cc:407] Connection header received
0619 02:34:26.161843 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:26.161846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:26.161911 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:26.162021 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:26.163623 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.164191 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.165430 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.165856 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.165993 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:26.168762 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:26.168766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:26.168769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:26.168800 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:26.170396 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:26.170966 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:26.170970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:26.170972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:26.171059 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:26.171485 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:26.171488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:26.171490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:26.171689 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:26.171804 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:26.172439 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 02:34:26.172582 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:34:26.173815 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:26.159404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44199 (local address 127.7.194.194:45579)
0619 02:34:26.159844 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:26.159916 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:26.161588 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:26.162198 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:26.162206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:26.162715 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:26.163452 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:26.163477 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.164320 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.164324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:26.165281 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:26.165290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.166039 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.166042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:26.166220 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:26.166886 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:26.166919 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:26.168570 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:26.170543 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:26.170549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:26.170567 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:26.170844 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:26.171162 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:26.171165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:26.171167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:26.171367 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:26.171800 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:26.171808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:26.172138 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 02:34:26.172385 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 02:34:26.172764 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260619 02:34:26.175158 8324 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44199
I20260619 02:34:26.175455 8324 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:26.175995 8324 heartbeater.cc:511] Master 127.7.194.254:44199 requested a full tablet report, sending...
I20260619 02:34:26.177137 7999 ts_manager.cc:194] Registered new tserver with Master: 750abb2b4cc243018c04bdb18adb1bcb (127.7.194.194:36237)
I20260619 02:34:26.177815 7999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:45579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:34:26.189293 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44199
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:38939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:26.303484 8329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:26.303790 8329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:26.303838 8329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:26.308010 8329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:26.308079 8329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:26.308141 8329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:26.313647 8329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:38939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:26.314766 8329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:26.315927 8329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:26.326223 8337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:26.326228 8334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:26.326231 8335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:26.326607 8329 server_base.cc:1034] running on GCE node
I20260619 02:34:26.327051 8329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:26.327749 8329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:26.328943 8329 hybrid_clock.cc:648] HybridClock initialized: now 1781836466328913 us; error 51 us; skew 500 ppm
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836466, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:26.331784 8329 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:34:26.332913 8329 webserver.cc:460] Webserver started at http://127.7.194.195:41097/ using document root <none> and password file <none>
I20260619 02:34:26.333636 8329 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:26.333732 8329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:26.334036 8329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:26.336176 8329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "7a78b71d99844a2f85a62a6770c63c88"
format_stamp: "Formatted at 2026-06-19 02:34:26 on dist-test-slave-9tkz"
server_key: "62c1be5fbe26a3a3537f37f3a383f6c0"
server_key_iv: "c693f7495875fa60e5a919d101175156"
server_key_version: "encryptionkey@0"
I20260619 02:34:26.336776 8329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "7a78b71d99844a2f85a62a6770c63c88"
format_stamp: "Formatted at 2026-06-19 02:34:26 on dist-test-slave-9tkz"
server_key: "62c1be5fbe26a3a3537f37f3a383f6c0"
server_key_iv: "c693f7495875fa60e5a919d101175156"
server_key_version: "encryptionkey@0"
I20260619 02:34:26.340772 8329 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:34:26.343324 8344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:26.344559 8329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:34:26.344722 8329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "7a78b71d99844a2f85a62a6770c63c88"
format_stamp: "Formatted at 2026-06-19 02:34:26 on dist-test-slave-9tkz"
server_key: "62c1be5fbe26a3a3537f37f3a383f6c0"
server_key_iv: "c693f7495875fa60e5a919d101175156"
server_key_version: "encryptionkey@0"
I20260619 02:34:26.344844 8329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:26.356389 8329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:26.359517 8329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:26.359736 8329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:26.360381 8329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:26.361258 8329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:26.361338 8329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:26.361406 8329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:26.361459 8329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:26.370903 8329 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:41763
I20260619 02:34:26.370926 8457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:41763 every 8 connection(s)
I20260619 02:34:26.371860 8329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:26.376118 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 8329
I20260619 02:34:26.376232 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:26.376502 7947 external_mini_cluster.cc:1428] Setting key 48eb9475940c898979551dd989a9dcea
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836466, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:26.387310 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:26.373960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:54045 (local address 127.7.194.254:44199)
0619 02:34:26.374113 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:26.374117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:26.375040 (+ 923us) server_negotiation.cc:407] Connection header received
0619 02:34:26.376724 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:26.376727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:26.376782 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:26.376870 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:26.378414 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.379030 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.380194 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.380413 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.380493 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:26.382956 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:26.382960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:26.382962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:26.382987 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:26.384681 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:26.385345 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:26.385347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:26.385349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:26.385438 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:26.385987 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:26.385991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:26.385992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:26.386273 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:26.386379 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:26.386981 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 02:34:26.387120 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:34:26.388226 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:26.374399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44199 (local address 127.7.194.195:54045)
0619 02:34:26.374861 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:26.374919 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:26.376483 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:26.377112 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:26.377119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:26.377596 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:26.378243 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:26.378259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.379150 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.379155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:26.380087 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:26.380095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.380542 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.380545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:26.380687 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:26.381270 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:26.381285 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:26.382804 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:26.384792 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:26.384798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:26.384816 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:26.385233 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:26.385654 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:26.385659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:26.385663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:26.385871 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:26.386373 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:26.386379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:26.386675 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 02:34:26.386909 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 02:34:26.387325 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":125,"threads_started":1}
I20260619 02:34:26.389405 8458 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44199
I20260619 02:34:26.389712 8458 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:26.390400 8458 heartbeater.cc:511] Master 127.7.194.254:44199 requested a full tablet report, sending...
I20260619 02:34:26.391594 7999 ts_manager.cc:194] Registered new tserver with Master: 7a78b71d99844a2f85a62a6770c63c88 (127.7.194.195:41763)
I20260619 02:34:26.392181 7999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:54045
I20260619 02:34:26.403251 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836466, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836466, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:26.430516 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:26.420147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.7.194.193:46593)
0619 02:34:26.420473 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:26.420493 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:26.420528 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:34:26.420718 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:26.420732 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:26.420958 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:26.421133 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:26.421897 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.422692 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.423611 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.424036 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.424180 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:26.426909 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:26.426918 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:26.426933 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:26.426959 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:26.428515 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:26.428981 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:26.428988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:26.428989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:26.429072 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:26.429385 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:26.429388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:26.429390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:26.429722 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:26.429981 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:26.430142 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 02:34:26.430308 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":114,"threads_started":1}
W20260619 02:34:26.431684 8104 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54062
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:26.457710 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:26.449491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.7.194.193:46593)
0619 02:34:26.449687 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:26.449691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:26.449711 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:34:26.449974 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:26.449977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:26.450025 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:26.450123 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:26.450834 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.451366 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.452139 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:26.452376 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:26.452452 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:26.454987 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:26.454990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:26.454992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:26.455018 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:26.456246 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:26.456629 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:26.456631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:26.456633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:26.456703 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:26.456925 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:26.456927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:26.456928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:26.457085 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:26.457177 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:26.457373 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 02:34:26.457485 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 02:34:26.458801 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 8061
I20260619 02:34:26.464653 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 8195
I20260619 02:34:26.470693 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 8329
I20260619 02:34:26.476284 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 7967
2026-06-19T02:34:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2201 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[8472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[8472](info): set up 2 sockets
Jun 19 02:34:26 dist-test-slave-9tkz krb5kdc[8472](info): commencing operation
krb5kdc: starting...
W20260619 02:34:27.513460 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:27 dist-test-slave-9tkz krb5kdc[8472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:27.636505 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:39861
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:39861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:27.749404 8485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:27.749786 8485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:27.749876 8485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:27.754061 8485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:27.754186 8485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:27.754227 8485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:27.754266 8485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:27.754297 8485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:27.759847 8485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:39861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:39861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:27.761129 8485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:27.762331 8485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:27.771448 8491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:27.771464 8493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:27.771464 8490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:27.771751 8485 server_base.cc:1034] running on GCE node
I20260619 02:34:27.772392 8485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:27.773676 8485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:27.774874 8485 hybrid_clock.cc:648] HybridClock initialized: now 1781836467774856 us; error 44 us; skew 500 ppm
Jun 19 02:34:27 dist-test-slave-9tkz krb5kdc[8472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836467, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:27.777786 8485 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:27.778796 8485 webserver.cc:460] Webserver started at http://127.7.194.254:35729/ using document root <none> and password file <none>
I20260619 02:34:27.779439 8485 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:27.779538 8485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:27.779757 8485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:27.781986 8485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "26df2d416258440593946e0fa182f7e8"
format_stamp: "Formatted at 2026-06-19 02:34:27 on dist-test-slave-9tkz"
server_key: "77c5a28768c0c919389ac4ffcde04de9"
server_key_iv: "670a8380831923cad8cb02d9fb44e95d"
server_key_version: "encryptionkey@0"
I20260619 02:34:27.782578 8485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "26df2d416258440593946e0fa182f7e8"
format_stamp: "Formatted at 2026-06-19 02:34:27 on dist-test-slave-9tkz"
server_key: "77c5a28768c0c919389ac4ffcde04de9"
server_key_iv: "670a8380831923cad8cb02d9fb44e95d"
server_key_version: "encryptionkey@0"
I20260619 02:34:27.786540 8485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:34:27.789124 8500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:27.790518 8485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:34:27.790686 8485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "26df2d416258440593946e0fa182f7e8"
format_stamp: "Formatted at 2026-06-19 02:34:27 on dist-test-slave-9tkz"
server_key: "77c5a28768c0c919389ac4ffcde04de9"
server_key_iv: "670a8380831923cad8cb02d9fb44e95d"
server_key_version: "encryptionkey@0"
I20260619 02:34:27.790813 8485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:27.807137 8485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:27.810316 8485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:27.810608 8485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:27.818089 8485 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:39861
I20260619 02:34:27.818110 8552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:39861 every 8 connection(s)
I20260619 02:34:27.819109 8485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:27.822017 8553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:27.823081 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 8485
I20260619 02:34:27.823163 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:27.823450 7947 external_mini_cluster.cc:1428] Setting key 5def88ad42eae33312b0eed5e7ca67c3
I20260619 02:34:27.828148 8553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8: Bootstrap starting.
I20260619 02:34:27.830562 8553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:27.831326 8553 log.cc:824] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:34:27 dist-test-slave-9tkz krb5kdc[8472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:27.833632 8553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8: No bootstrap required, opened a new log
I20260619 02:34:27.837858 8553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df2d416258440593946e0fa182f7e8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39861 } }
I20260619 02:34:27.838127 8553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:27.838173 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:27.824975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.7.194.254:39861)
0619 02:34:27.825420 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:27.825436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:27.825472 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:34:27.826166 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:27.826185 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:27.826541 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:27.826898 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:27.827725 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:27.828776 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:27.829720 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:27.830201 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:27.830383 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:27.833096 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:27.833102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:27.833120 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:27.833151 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:27.835092 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:27.835649 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:27.835653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:27.835657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:27.835780 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:27.836104 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:27.836106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:27.836107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:27.836518 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:27.836668 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:27.836963 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 02:34:27.837280 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
I20260619 02:34:27.838213 8553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df2d416258440593946e0fa182f7e8, State: Initialized, Role: FOLLOWER
I20260619 02:34:27.838725 8553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df2d416258440593946e0fa182f7e8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39861 } }
I20260619 02:34:27.838879 8553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:27.838941 8553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:27.839043 8553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:27.840046 8553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df2d416258440593946e0fa182f7e8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39861 } }
I20260619 02:34:27.840453 8553 leader_election.cc:302] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df2d416258440593946e0fa182f7e8; no voters:
I20260619 02:34:27.840780 8553 leader_election.cc:288] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:27.840895 8558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:27.841125 8558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 26df2d416258440593946e0fa182f7e8, State: Running, Role: LEADER
I20260619 02:34:27.841454 8558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df2d416258440593946e0fa182f7e8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39861 } }
I20260619 02:34:27.842088 8553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:27.843267 8560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26df2d416258440593946e0fa182f7e8. Latest consensus state: current_term: 1 leader_uuid: "26df2d416258440593946e0fa182f7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df2d416258440593946e0fa182f7e8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39861 } } }
I20260619 02:34:27.843412 8560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:27.843268 8559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26df2d416258440593946e0fa182f7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df2d416258440593946e0fa182f7e8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39861 } } }
I20260619 02:34:27.843731 8559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:27.843799 8567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:27.847057 8567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:27.853302 8567 catalog_manager.cc:1347] Generated new cluster ID: bf4bea32f4564a599a61a095304c7891
I20260619 02:34:27.853380 8567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:27.873695 8567 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:27.875130 8567 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:27.891520 8567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26df2d416258440593946e0fa182f7e8: Generated new TSK 0
I20260619 02:34:27.892455 8567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:34:27.925050 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:39861
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:28.045905 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:28.046214 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:28.046268 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:28.050539 8579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:28.050626 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:28.050699 8579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:28.056280 8579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:39861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:28.057431 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:28.058604 8579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:28.069105 8587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:28.069195 8579 server_base.cc:1034] running on GCE node
W20260619 02:34:28.069105 8585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:28.069160 8584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:28.069942 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:28.070623 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:28.071852 8579 hybrid_clock.cc:648] HybridClock initialized: now 1781836468071834 us; error 29 us; skew 500 ppm
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:28.074770 8579 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:34:28.075732 8579 webserver.cc:460] Webserver started at http://127.7.194.193:34969/ using document root <none> and password file <none>
I20260619 02:34:28.076352 8579 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:28.076457 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:28.076695 8579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:28.078897 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "c3e0435105c84d589d4a8541e267dc64"
format_stamp: "Formatted at 2026-06-19 02:34:28 on dist-test-slave-9tkz"
server_key: "b7611ba5a6a18ba525ba6a0d7e1b3734"
server_key_iv: "58ea80139656bf7b19e8681e3d46499e"
server_key_version: "encryptionkey@0"
I20260619 02:34:28.079499 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "c3e0435105c84d589d4a8541e267dc64"
format_stamp: "Formatted at 2026-06-19 02:34:28 on dist-test-slave-9tkz"
server_key: "b7611ba5a6a18ba525ba6a0d7e1b3734"
server_key_iv: "58ea80139656bf7b19e8681e3d46499e"
server_key_version: "encryptionkey@0"
I20260619 02:34:28.083467 8579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:28.086001 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:28.087261 8579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:34:28.087409 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "c3e0435105c84d589d4a8541e267dc64"
format_stamp: "Formatted at 2026-06-19 02:34:28 on dist-test-slave-9tkz"
server_key: "b7611ba5a6a18ba525ba6a0d7e1b3734"
server_key_iv: "58ea80139656bf7b19e8681e3d46499e"
server_key_version: "encryptionkey@0"
I20260619 02:34:28.087529 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:28.104061 8579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:28.107199 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:28.107429 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:28.108094 8579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:28.108955 8579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:28.109028 8579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:28.109094 8579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:28.109128 8579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:28.118561 8579 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:44997
I20260619 02:34:28.118583 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:44997 every 8 connection(s)
I20260619 02:34:28.119516 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:28.122443 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 8579
I20260619 02:34:28.122568 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:28.122864 7947 external_mini_cluster.cc:1428] Setting key 9d4b318f8c8ba18f0f90402754311d1e
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:34:28.136039 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.121448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:46009 (local address 127.7.194.254:39861)
0619 02:34:28.121624 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.121633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.122617 (+ 984us) server_negotiation.cc:407] Connection header received
0619 02:34:28.124387 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.124390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.124449 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.124551 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:28.126369 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.127279 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.128498 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.128950 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.129078 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.131825 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:28.131830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:28.131836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:28.131871 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:28.133676 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.134320 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.134323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.134324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.134402 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.134729 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.134731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.134732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.134927 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:28.135004 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.135650 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.135807 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:34:28.136960 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.121890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:39861 (local address 127.7.194.193:46009)
0619 02:34:28.122437 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:28.122496 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:28.124097 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:28.124750 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:28.124760 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:28.125358 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:28.126162 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.126186 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.127428 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.127431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.128363 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.128372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.129137 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.129140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.129319 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.130070 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:28.130093 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:28.131637 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:28.133829 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:28.133834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:28.133848 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:28.134197 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:28.134490 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:28.134492 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:28.134494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:28.134641 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:28.135030 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:28.135036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:28.135402 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 02:34:28.135572 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 02:34:28.135998 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":111,"threads_started":1}
I20260619 02:34:28.138516 8708 heartbeater.cc:347] Connected to a master server at 127.7.194.254:39861
I20260619 02:34:28.138799 8708 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:28.139361 8708 heartbeater.cc:511] Master 127.7.194.254:39861 requested a full tablet report, sending...
I20260619 02:34:28.140990 8517 ts_manager.cc:194] Registered new tserver with Master: c3e0435105c84d589d4a8541e267dc64 (127.7.194.193:44997)
I20260619 02:34:28.142429 8517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:46009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:34:28.153118 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:39861
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:28.267581 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:28.267877 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:28.267925 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:28.271993 8713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:28.272060 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:28.272169 8713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:28.277515 8713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:39861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:28.278725 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:28.279821 8713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:28.290033 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:28.290151 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:28.290128 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:28.290126 8713 server_base.cc:1034] running on GCE node
I20260619 02:34:28.290797 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:28.291474 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:28.292676 8713 hybrid_clock.cc:648] HybridClock initialized: now 1781836468292640 us; error 49 us; skew 500 ppm
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:28.295665 8713 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:34:28.296576 8713 webserver.cc:460] Webserver started at http://127.7.194.194:40307/ using document root <none> and password file <none>
I20260619 02:34:28.297122 8713 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:28.297183 8713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:28.297364 8713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:28.299610 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "f4bd67b969ab42b781039d4074fa8853"
format_stamp: "Formatted at 2026-06-19 02:34:28 on dist-test-slave-9tkz"
server_key: "06b7f141d06b5b31b02abc8fd0a30c9e"
server_key_iv: "67d8a752f2d7b713de90dd1a2e377eda"
server_key_version: "encryptionkey@0"
I20260619 02:34:28.300163 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "f4bd67b969ab42b781039d4074fa8853"
format_stamp: "Formatted at 2026-06-19 02:34:28 on dist-test-slave-9tkz"
server_key: "06b7f141d06b5b31b02abc8fd0a30c9e"
server_key_iv: "67d8a752f2d7b713de90dd1a2e377eda"
server_key_version: "encryptionkey@0"
I20260619 02:34:28.304255 8713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:28.306849 8728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:28.308143 8713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:28.308292 8713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "f4bd67b969ab42b781039d4074fa8853"
format_stamp: "Formatted at 2026-06-19 02:34:28 on dist-test-slave-9tkz"
server_key: "06b7f141d06b5b31b02abc8fd0a30c9e"
server_key_iv: "67d8a752f2d7b713de90dd1a2e377eda"
server_key_version: "encryptionkey@0"
I20260619 02:34:28.308408 8713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:28.330961 8713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:28.334136 8713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:28.334349 8713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:28.334982 8713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:28.335850 8713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:28.335896 8713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:28.335937 8713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:28.335997 8713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:28.345400 8713 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:43685
I20260619 02:34:28.345480 8841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:43685 every 8 connection(s)
I20260619 02:34:28.346390 8713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:28.350430 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 8713
I20260619 02:34:28.350564 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:28.350802 7947 external_mini_cluster.cc:1428] Setting key 2c9ddb6bfa41711b9a0096a5fa8926b4
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:34:28.363294 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.348298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:53901 (local address 127.7.194.254:39861)
0619 02:34:28.348458 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.348461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.349333 (+ 872us) server_negotiation.cc:407] Connection header received
0619 02:34:28.351030 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.351033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.351095 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.351197 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:28.352878 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.353798 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.355659 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.355932 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.356033 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.359322 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:28.359326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:28.359328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:28.359353 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:28.360881 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.361594 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.361598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.361600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.361681 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.362055 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.362061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.362065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.362260 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:28.362373 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.362977 (+ 604us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.363105 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:34:28.364168 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.348704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:39861 (local address 127.7.194.194:53901)
0619 02:34:28.349160 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:28.349204 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:28.350732 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:28.351423 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:28.351434 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:28.352006 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:28.352694 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.352714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.353973 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.353976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.355519 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.355530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.356108 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.356115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.356375 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.357144 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:28.357168 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:28.359126 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:28.361032 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:28.361041 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:28.361059 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:28.361449 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:28.361784 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:28.361788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:28.361790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:28.361948 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:28.362386 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:28.362403 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:28.362668 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 02:34:28.362937 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 02:34:28.363303 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:34:28.365451 8842 heartbeater.cc:347] Connected to a master server at 127.7.194.254:39861
I20260619 02:34:28.365737 8842 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:28.366357 8842 heartbeater.cc:511] Master 127.7.194.254:39861 requested a full tablet report, sending...
I20260619 02:34:28.367508 8517 ts_manager.cc:194] Registered new tserver with Master: f4bd67b969ab42b781039d4074fa8853 (127.7.194.194:43685)
I20260619 02:34:28.368407 8517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:53901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:34:28.380846 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:39861
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:28.498472 8847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:28.498749 8847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:28.498867 8847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:28.502975 8847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:28.503044 8847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:28.503159 8847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:28.508761 8847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:39861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:28.510002 8847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:28.511102 8847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:28.521240 8852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:28.521359 8853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:28.521447 8855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:28.521605 8847 server_base.cc:1034] running on GCE node
I20260619 02:34:28.521981 8847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:28.522614 8847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:28.523789 8847 hybrid_clock.cc:648] HybridClock initialized: now 1781836468523755 us; error 47 us; skew 500 ppm
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:28.526832 8847 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:34:28.527792 8847 webserver.cc:460] Webserver started at http://127.7.194.195:34631/ using document root <none> and password file <none>
I20260619 02:34:28.528386 8847 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:28.528448 8847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:28.528702 8847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:28.530964 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "4affb13a6cb44e1f942d10b72e1823ad"
format_stamp: "Formatted at 2026-06-19 02:34:28 on dist-test-slave-9tkz"
server_key: "2b20ab1c86b6ccf53e715f7b856f08c0"
server_key_iv: "999ca4bb6dbe4f3285f3d135cd2e9d05"
server_key_version: "encryptionkey@0"
I20260619 02:34:28.531548 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "4affb13a6cb44e1f942d10b72e1823ad"
format_stamp: "Formatted at 2026-06-19 02:34:28 on dist-test-slave-9tkz"
server_key: "2b20ab1c86b6ccf53e715f7b856f08c0"
server_key_iv: "999ca4bb6dbe4f3285f3d135cd2e9d05"
server_key_version: "encryptionkey@0"
I20260619 02:34:28.535562 8847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:28.538359 8862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:28.539767 8847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:28.539944 8847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "4affb13a6cb44e1f942d10b72e1823ad"
format_stamp: "Formatted at 2026-06-19 02:34:28 on dist-test-slave-9tkz"
server_key: "2b20ab1c86b6ccf53e715f7b856f08c0"
server_key_iv: "999ca4bb6dbe4f3285f3d135cd2e9d05"
server_key_version: "encryptionkey@0"
I20260619 02:34:28.540067 8847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:28.566624 8847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:28.570325 8847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:28.570577 8847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:28.571214 8847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:28.572106 8847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:28.572162 8847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:28.572248 8847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:28.572289 8847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:28.581424 8847 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:37413
I20260619 02:34:28.581494 8975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:37413 every 8 connection(s)
I20260619 02:34:28.582412 8847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:28.587713 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 8847
I20260619 02:34:28.587858 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:28.588138 7947 external_mini_cluster.cc:1428] Setting key 010a8136ac9ce6df145b7551af4522ea
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:28.598169 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.584973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:57677 (local address 127.7.194.254:39861)
0619 02:34:28.585168 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.585175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.586311 (+ 1136us) server_negotiation.cc:407] Connection header received
0619 02:34:28.588073 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.588076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.588134 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.588218 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:28.589838 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.590521 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.591739 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.591983 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.592101 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.594698 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:28.594702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:28.594704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:28.594728 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:28.596107 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.596577 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.596580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.596581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.596657 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.596998 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.597001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.597002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.597266 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:28.597385 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.597941 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.598040 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:34:28.599038 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.585561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:39861 (local address 127.7.194.195:57677)
0619 02:34:28.586122 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:28.586180 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:28.587840 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:28.588400 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:28.588411 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:28.588946 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:28.589694 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.589711 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.590648 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.590651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.591616 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.591624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.592135 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.592138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.592304 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.592942 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:28.592958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:28.594533 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:28.596212 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:28.596216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:28.596229 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:28.596482 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:28.596731 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:28.596733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:28.596734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:28.596860 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:28.597374 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:28.597380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:28.597693 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 02:34:28.597849 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 02:34:28.598112 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":107,"threads_started":1}
I20260619 02:34:28.600241 8976 heartbeater.cc:347] Connected to a master server at 127.7.194.254:39861
I20260619 02:34:28.600447 8976 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:28.601023 8976 heartbeater.cc:511] Master 127.7.194.254:39861 requested a full tablet report, sending...
I20260619 02:34:28.602195 8517 ts_manager.cc:194] Registered new tserver with Master: 4affb13a6cb44e1f942d10b72e1823ad (127.7.194.195:37413)
I20260619 02:34:28.602676 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:34:28.602905 8517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:57677
I20260619 02:34:28.614426 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.606583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.7.194.254:39861)
0619 02:34:28.606740 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.606745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.606950 (+ 205us) server_negotiation.cc:407] Connection header received
0619 02:34:28.607105 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.607108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.607156 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.607244 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:28.607951 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.608474 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.609309 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.609537 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.609735 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.611070 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:28.611073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:28.611076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:28.611101 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:28.612573 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.613092 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.613098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.613100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.613183 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.613463 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.613466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.613468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.613742 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:28.613887 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.614133 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.614251 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:34:28.619343 8517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:34:28.622697 8517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:28.637487 8988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.631401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:37413 (local address 127.0.0.1:56002)
0619 02:34:28.631884 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:28.631896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:28.631963 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:28.633298 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:28.633301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:28.633320 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:28.633473 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.633479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.634825 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.634828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.636102 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.636113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.637118 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.637124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.637213 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.637232 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:34:28.637282 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:34:28.637328 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":54,"spinlock_wait_cycles":11520,"thread_start_us":114,"threads_started":1}
I20260619 02:34:28.637502 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.631528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.7.194.193:44997)
0619 02:34:28.632235 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.632251 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.632277 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:34:28.632352 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.632357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.632593 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.632751 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:28.633246 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.634466 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.636019 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.636813 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.636926 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.637083 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.637222 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.637330 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":77,"threads_started":1}
I20260619 02:34:28.637534 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.632217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.7.194.195:37413)
0619 02:34:28.632749 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.632761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.632785 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:34:28.632857 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.632861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.633048 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.633196 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:28.633616 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.634683 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.636231 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.636994 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.637090 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.637259 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.637317 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.637393 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":149,"threads_started":1}
I20260619 02:34:28.637876 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.631378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:44997 (local address 127.0.0.1:50798)
0619 02:34:28.631756 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:28.631814 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:28.631964 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:28.632944 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:28.632950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:28.632973 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:28.633109 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.633119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.634610 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.634613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.635878 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.635886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.636929 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.636932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.637005 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.637035 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:34:28.637174 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 02:34:28.637303 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":114,"thread_start_us":81,"threads_started":1}
I20260619 02:34:28.639099 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.632141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:43685 (local address 127.0.0.1:51920)
0619 02:34:28.632448 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:28.632462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:28.632552 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:28.633262 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:28.633268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:28.633291 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:28.633465 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.633473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.634586 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.634593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.636102 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.636113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.638685 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.638689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.638783 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.638815 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 02:34:28.638866 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:34:28.638952 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":138,"spinlock_wait_cycles":8576,"thread_start_us":92,"threads_started":1}
I20260619 02:34:28.639642 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.632309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.7.194.194:43685)
0619 02:34:28.632799 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.632811 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.632832 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:34:28.632908 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.632914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.633113 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.633643 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:28.633769 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.634450 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.637716 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.638545 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.639289 (+ 744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.639414 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.639456 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.639527 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":129,"threads_started":1}
I20260619 02:34:28.640918 8642 tablet_service.cc:1467] Processing CreateTablet for tablet f9c0bef303e24d6db2c6fd48d157419e (DEFAULT_TABLE table=test-table [id=298274a3a8a8431cac2c1a292818c6f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:28.640923 8910 tablet_service.cc:1467] Processing CreateTablet for tablet f9c0bef303e24d6db2c6fd48d157419e (DEFAULT_TABLE table=test-table [id=298274a3a8a8431cac2c1a292818c6f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:28.642117 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c0bef303e24d6db2c6fd48d157419e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:28.642117 8910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c0bef303e24d6db2c6fd48d157419e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:28.642264 8776 tablet_service.cc:1467] Processing CreateTablet for tablet f9c0bef303e24d6db2c6fd48d157419e (DEFAULT_TABLE table=test-table [id=298274a3a8a8431cac2c1a292818c6f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:28.643358 8776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c0bef303e24d6db2c6fd48d157419e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:28.649533 8993 tablet_bootstrap.cc:492] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853: Bootstrap starting.
I20260619 02:34:28.650393 8994 tablet_bootstrap.cc:492] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64: Bootstrap starting.
I20260619 02:34:28.652606 8993 tablet_bootstrap.cc:654] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:28.652621 8994 tablet_bootstrap.cc:654] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:28.653193 8995 tablet_bootstrap.cc:492] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad: Bootstrap starting.
I20260619 02:34:28.653532 8993 log.cc:824] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:28.653532 8994 log.cc:824] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:28.655642 8995 tablet_bootstrap.cc:654] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:28.655694 8994 tablet_bootstrap.cc:492] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64: No bootstrap required, opened a new log
I20260619 02:34:28.655694 8993 tablet_bootstrap.cc:492] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853: No bootstrap required, opened a new log
I20260619 02:34:28.655872 8993 ts_tablet_manager.cc:1397] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260619 02:34:28.655872 8994 ts_tablet_manager.cc:1397] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 02:34:28.656445 8995 log.cc:824] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:28.658265 8995 tablet_bootstrap.cc:492] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad: No bootstrap required, opened a new log
I20260619 02:34:28.658485 8995 ts_tablet_manager.cc:1397] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:34:28.659968 8993 raft_consensus.cc:348] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } }
I20260619 02:34:28.660259 8993 raft_consensus.cc:374] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:28.660313 8993 raft_consensus.cc:729] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bd67b969ab42b781039d4074fa8853, State: Initialized, Role: FOLLOWER
I20260619 02:34:28.660199 8994 raft_consensus.cc:348] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } }
I20260619 02:34:28.660419 8994 raft_consensus.cc:374] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:28.660459 8994 raft_consensus.cc:729] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e0435105c84d589d4a8541e267dc64, State: Initialized, Role: FOLLOWER
I20260619 02:34:28.660837 8993 consensus_queue.cc:260] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } }
I20260619 02:34:28.660837 8994 consensus_queue.cc:260] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } }
I20260619 02:34:28.661761 8708 heartbeater.cc:503] Master 127.7.194.254:39861 was elected leader, sending a full tablet report...
I20260619 02:34:28.661897 8994 ts_tablet_manager.cc:1428] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260619 02:34:28.661887 8993 ts_tablet_manager.cc:1428] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260619 02:34:28.662213 8842 heartbeater.cc:503] Master 127.7.194.254:39861 was elected leader, sending a full tablet report...
I20260619 02:34:28.662047 8995 raft_consensus.cc:348] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } }
I20260619 02:34:28.662290 8995 raft_consensus.cc:374] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:28.662350 8995 raft_consensus.cc:729] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4affb13a6cb44e1f942d10b72e1823ad, State: Initialized, Role: FOLLOWER
I20260619 02:34:28.662772 8995 consensus_queue.cc:260] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } }
I20260619 02:34:28.663443 8976 heartbeater.cc:503] Master 127.7.194.254:39861 was elected leader, sending a full tablet report...
I20260619 02:34:28.663846 8995 ts_tablet_manager.cc:1428] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:34:28.707701 9001 raft_consensus.cc:482] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:28.707968 9001 raft_consensus.cc:504] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } }
I20260619 02:34:28.709201 9001 leader_election.cc:288] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e0435105c84d589d4a8541e267dc64 (127.7.194.193:44997), f4bd67b969ab42b781039d4074fa8853 (127.7.194.194:43685)
I20260619 02:34:28.713410 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.709426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:43685 (local address 127.7.194.195:58937)
0619 02:34:28.709528 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:28.709571 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:28.709685 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:28.709991 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:28.709994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:28.710014 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:28.710114 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.710119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.711080 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.711083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.712117 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.712128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.712971 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.712975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.713081 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.713106 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:34:28.713185 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:34:28.713253 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 02:34:28.713420 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.709582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:58937 (local address 127.7.194.194:43685)
0619 02:34:28.709728 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.709735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.709752 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:34:28.709805 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.709808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.709866 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.709959 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:28.710234 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.710974 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.712241 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.712863 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.712953 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.712995 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.713249 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.713312 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:34:28.714167 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.709564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:36075 (local address 127.7.194.193:44997)
0619 02:34:28.709719 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.709722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.710343 (+ 621us) server_negotiation.cc:407] Connection header received
0619 02:34:28.710436 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.710438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.710493 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.710557 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:28.710833 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.711485 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.712834 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.713483 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.713628 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.713693 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.714015 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.714079 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:34:28.714190 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.709426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:44997 (local address 127.7.194.195:36075)
0619 02:34:28.710230 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:28.710243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:28.710340 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:28.710592 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:28.710595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:28.710611 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:28.710721 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.710726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.711590 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.711593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.712661 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:28.712671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.713762 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.713765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:28.713881 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.713907 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:34:28.713987 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:34:28.714048 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":330,"thread_start_us":134,"threads_started":1}
I20260619 02:34:28.714604 8796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c0bef303e24d6db2c6fd48d157419e" candidate_uuid: "4affb13a6cb44e1f942d10b72e1823ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bd67b969ab42b781039d4074fa8853" is_pre_election: true
I20260619 02:34:28.714937 8796 raft_consensus.cc:2454] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4affb13a6cb44e1f942d10b72e1823ad in term 0.
I20260619 02:34:28.715250 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c0bef303e24d6db2c6fd48d157419e" candidate_uuid: "4affb13a6cb44e1f942d10b72e1823ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e0435105c84d589d4a8541e267dc64" is_pre_election: true
I20260619 02:34:28.715487 8863 leader_election.cc:302] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4affb13a6cb44e1f942d10b72e1823ad, f4bd67b969ab42b781039d4074fa8853; no voters:
I20260619 02:34:28.715569 8662 raft_consensus.cc:2454] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4affb13a6cb44e1f942d10b72e1823ad in term 0.
I20260619 02:34:28.715832 9001 raft_consensus.cc:2781] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:28.715960 9001 raft_consensus.cc:482] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:28.716015 9001 raft_consensus.cc:3037] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:28.717104 9001 raft_consensus.cc:504] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } }
I20260619 02:34:28.717497 9001 leader_election.cc:288] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [CANDIDATE]: Term 1 election: Requested vote from peers c3e0435105c84d589d4a8541e267dc64 (127.7.194.193:44997), f4bd67b969ab42b781039d4074fa8853 (127.7.194.194:43685)
I20260619 02:34:28.717936 8796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c0bef303e24d6db2c6fd48d157419e" candidate_uuid: "4affb13a6cb44e1f942d10b72e1823ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bd67b969ab42b781039d4074fa8853"
I20260619 02:34:28.718070 8796 raft_consensus.cc:3037] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:28.718145 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c0bef303e24d6db2c6fd48d157419e" candidate_uuid: "4affb13a6cb44e1f942d10b72e1823ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e0435105c84d589d4a8541e267dc64"
I20260619 02:34:28.718279 8662 raft_consensus.cc:3037] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:28.719199 8796 raft_consensus.cc:2454] T f9c0bef303e24d6db2c6fd48d157419e P f4bd67b969ab42b781039d4074fa8853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4affb13a6cb44e1f942d10b72e1823ad in term 1.
I20260619 02:34:28.719368 8662 raft_consensus.cc:2454] T f9c0bef303e24d6db2c6fd48d157419e P c3e0435105c84d589d4a8541e267dc64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4affb13a6cb44e1f942d10b72e1823ad in term 1.
I20260619 02:34:28.719552 8863 leader_election.cc:302] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4affb13a6cb44e1f942d10b72e1823ad, f4bd67b969ab42b781039d4074fa8853; no voters:
I20260619 02:34:28.719736 9001 raft_consensus.cc:2781] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:28.719985 9001 raft_consensus.cc:686] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [term 1 LEADER]: Becoming Leader. State: Replica: 4affb13a6cb44e1f942d10b72e1823ad, State: Running, Role: LEADER
I20260619 02:34:28.720335 9001 consensus_queue.cc:237] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } }
I20260619 02:34:28.723829 8516 catalog_manager.cc:5495] T f9c0bef303e24d6db2c6fd48d157419e P 4affb13a6cb44e1f942d10b72e1823ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 4affb13a6cb44e1f942d10b72e1823ad (127.7.194.195). New cstate: current_term: 1 leader_uuid: "4affb13a6cb44e1f942d10b72e1823ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e0435105c84d589d4a8541e267dc64" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4affb13a6cb44e1f942d10b72e1823ad" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4bd67b969ab42b781039d4074fa8853" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43685 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:28.738435 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.728963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.7.194.193:44997)
0619 02:34:28.729094 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.729098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.729265 (+ 167us) server_negotiation.cc:407] Connection header received
0619 02:34:28.729376 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.729379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.729431 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.729504 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:28.730226 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.730749 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.731565 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.731781 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.731862 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.734679 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:28.734690 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:28.734705 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:28.734730 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:28.736530 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.737027 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.737033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.737034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.737116 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.737430 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.737433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.737435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.737805 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:28.737983 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.738185 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.738306 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:28.769493 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.761235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.7.194.193:44997)
0619 02:34:28.761436 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.761440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.761473 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:34:28.761688 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.761691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.761738 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.761803 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:28.762456 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.762968 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.763767 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.763997 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.764077 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.766603 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:28.766606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:28.766608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:28.766631 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:28.767872 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.768278 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.768280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.768281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.768349 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.768626 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.768632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.768635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.768838 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:28.768975 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.769261 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.769365 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[8472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:28.796455 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:28.787213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.7.194.193:44997)
0619 02:34:28.787406 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:28.787410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:28.787427 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:34:28.787545 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:28.787548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:28.787593 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:28.787669 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:28.788411 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.788901 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.789910 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:28.790142 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:28.790231 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:28.793042 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:28.793045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:28.793048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:28.793073 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:28.794697 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.795140 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.795143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.795144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.795222 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:28.795564 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:28.795567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:28.795571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:28.795746 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:28.795910 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:28.796163 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 02:34:28.796281 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 02:34:28.797979 8622 tablet_service.cc:2908] Rejecting scan request for tablet f9c0bef303e24d6db2c6fd48d157419e: Uninitialized: clean time has not yet been initialized
I20260619 02:34:28.800262 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 8579
I20260619 02:34:28.806479 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 8713
I20260619 02:34:28.812266 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 8847
I20260619 02:34:28.817965 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 8485
2026-06-19T02:34:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2341 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[9017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[9017](info): set up 2 sockets
Jun 19 02:34:28 dist-test-slave-9tkz krb5kdc[9017](info): commencing operation
krb5kdc: starting...
W20260619 02:34:29.857370 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:29 dist-test-slave-9tkz krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:29.982519 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44621
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:44621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:30.098374 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:30.098646 9030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:30.098687 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:30.102775 9030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:30.102847 9030 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:34:30.102865 9030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:30.102882 9030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:30.102898 9030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:30.102914 9030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:30.108824 9030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:44621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:30.110067 9030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:30.111230 9030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:30.120424 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:30.120477 9035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:30.120430 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:30.120824 9030 server_base.cc:1034] running on GCE node
I20260619 02:34:30.121429 9030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:30.122596 9030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:30.123799 9030 hybrid_clock.cc:648] HybridClock initialized: now 1781836470123761 us; error 50 us; skew 500 ppm
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:30.130833 9030 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:30.131902 9030 webserver.cc:460] Webserver started at http://127.7.194.254:40655/ using document root <none> and password file <none>
I20260619 02:34:30.132525 9030 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:30.132588 9030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:30.132845 9030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:30.135161 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "9c5e3c439ce144529ccb003785afe685"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "07e563e3dd28ed9c26e0ac68f259b002"
server_key_iv: "9f701922e9c22fb8ddd24177667d3a75"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.135789 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "9c5e3c439ce144529ccb003785afe685"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "07e563e3dd28ed9c26e0ac68f259b002"
server_key_iv: "9f701922e9c22fb8ddd24177667d3a75"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.139797 9030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:30.142303 9045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:30.143677 9030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:34:30.143875 9030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "9c5e3c439ce144529ccb003785afe685"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "07e563e3dd28ed9c26e0ac68f259b002"
server_key_iv: "9f701922e9c22fb8ddd24177667d3a75"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.143998 9030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:30.155633 9030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:30.158649 9030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:30.158865 9030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:30.167233 9030 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:44621
I20260619 02:34:30.167240 9107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:44621 every 8 connection(s)
I20260619 02:34:30.168270 9030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:30.169516 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 9030
I20260619 02:34:30.169651 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:30.169938 7947 external_mini_cluster.cc:1428] Setting key 2dcf49c9f702c7b60cca8642d8739a28
I20260619 02:34:30.171800 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:30.178709 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685: Bootstrap starting.
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:30.181326 9108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:30.182502 9108 log.cc:824] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:30.184778 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685: No bootstrap required, opened a new log
I20260619 02:34:30.185375 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.171508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.7.194.254:44621)
0619 02:34:30.171934 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.171959 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.171995 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:34:30.172640 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.172658 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.173007 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.173304 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:30.174332 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.175363 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.176538 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.177054 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.177175 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.180182 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:30.180187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:30.180201 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:30.180236 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:30.182247 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.182780 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.182786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.182791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.182943 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.183233 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.183239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.183242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.183690 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:30.183848 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.184147 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.184482 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":154,"threads_started":1}
I20260619 02:34:30.187983 9108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5e3c439ce144529ccb003785afe685" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44621 } }
I20260619 02:34:30.188242 9108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:30.188324 9108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5e3c439ce144529ccb003785afe685, State: Initialized, Role: FOLLOWER
I20260619 02:34:30.188750 9108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5e3c439ce144529ccb003785afe685" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44621 } }
I20260619 02:34:30.188903 9108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:30.188994 9108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:30.189114 9108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:30.190146 9108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5e3c439ce144529ccb003785afe685" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44621 } }
I20260619 02:34:30.190512 9108 leader_election.cc:302] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c5e3c439ce144529ccb003785afe685; no voters:
I20260619 02:34:30.190828 9108 leader_election.cc:288] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:30.190980 9113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:30.191226 9113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [term 1 LEADER]: Becoming Leader. State: Replica: 9c5e3c439ce144529ccb003785afe685, State: Running, Role: LEADER
I20260619 02:34:30.191548 9113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5e3c439ce144529ccb003785afe685" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44621 } }
I20260619 02:34:30.192123 9108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:30.193302 9114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c5e3c439ce144529ccb003785afe685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5e3c439ce144529ccb003785afe685" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44621 } } }
I20260619 02:34:30.193440 9114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:30.193740 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c5e3c439ce144529ccb003785afe685. Latest consensus state: current_term: 1 leader_uuid: "9c5e3c439ce144529ccb003785afe685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5e3c439ce144529ccb003785afe685" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44621 } } }
I20260619 02:34:30.193848 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:30.194142 9122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:30.197326 9122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:30.203574 9122 catalog_manager.cc:1347] Generated new cluster ID: 2e929a7efb4d4156946675ad86d56fa3
I20260619 02:34:30.203686 9122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:30.215286 9122 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:30.216334 9122 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:30.220687 9122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685: Generated new TSK 0
I20260619 02:34:30.221444 9122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:34:30.252980 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44621
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:30.369801 9134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:30.370150 9134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:30.370196 9134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:30.374459 9134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:30.374528 9134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:30.374570 9134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:34:30.374603 9134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:30.380173 9134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.194.254:44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:30.381354 9134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:30.382522 9134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:30.392707 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:30.392727 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:30.392865 9134 server_base.cc:1034] running on GCE node
W20260619 02:34:30.392714 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:30.393517 9134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:30.394326 9134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:30.395561 9134 hybrid_clock.cc:648] HybridClock initialized: now 1781836470395545 us; error 28 us; skew 500 ppm
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:30.398674 9134 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:34:30.399654 9134 webserver.cc:460] Webserver started at http://127.7.194.193:39811/ using document root <none> and password file <none>
I20260619 02:34:30.400254 9134 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:30.400315 9134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:30.400528 9134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:30.403558 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "7c4c210e830147cfb524aa90c4d319d4"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "b563757827159df646f18b53eee7903e"
server_key_iv: "9e00695441f23fcb710e2cb42ce21c44"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.404300 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "7c4c210e830147cfb524aa90c4d319d4"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "b563757827159df646f18b53eee7903e"
server_key_iv: "9e00695441f23fcb710e2cb42ce21c44"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.408484 9134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:34:30.411234 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:30.412564 9134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:34:30.412712 9134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "7c4c210e830147cfb524aa90c4d319d4"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "b563757827159df646f18b53eee7903e"
server_key_iv: "9e00695441f23fcb710e2cb42ce21c44"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.412824 9134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:30.427147 9134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:30.430548 9134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:30.430708 9134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:30.432140 9134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:30.432191 9134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:30.432232 9134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:30.432304 9134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:30.443178 9134 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:41967
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:30.443235 9264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:41967 every 8 connection(s)
I20260619 02:34:30.444244 9134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:30.448005 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.433229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:55729 (local address 127.7.194.254:44621)
0619 02:34:30.433393 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.433397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.434512 (+ 1115us) server_negotiation.cc:407] Connection header received
0619 02:34:30.435870 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.435877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.435947 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.436051 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:30.437743 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.438434 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.440210 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.440464 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.440588 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.443964 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:30.443968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:30.443970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:30.443999 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:30.445933 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.446443 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.446445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.446446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.446535 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.446839 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.446841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.446842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.447038 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:30.447122 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.447741 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.447868 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:34:30.448926 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.433805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44621 (local address 127.7.194.193:55729)
0619 02:34:30.434327 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:30.434371 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:30.435547 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:30.436253 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:30.436260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:30.436782 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:30.437536 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.437578 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.438578 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.438581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.440072 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.440081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.440630 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.440634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.440908 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.441872 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:30.441892 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:30.443722 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:30.446057 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.446061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.446073 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.446337 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.446615 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.446617 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.446619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.446744 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.447170 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:30.447175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:30.447497 (+ 322us) client_negotiation.cc:769] Sending connection context
0619 02:34:30.447661 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 02:34:30.447958 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":137,"threads_started":1}
I20260619 02:34:30.449935 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 9134
I20260619 02:34:30.450063 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:30.450344 9265 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44621
I20260619 02:34:30.450383 7947 external_mini_cluster.cc:1428] Setting key 9f495f520d3fb7dc6cdba179c4cdba14
I20260619 02:34:30.450589 9265 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:30.451205 9265 heartbeater.cc:511] Master 127.7.194.254:44621 requested a full tablet report, sending...
I20260619 02:34:30.452960 9061 ts_manager.cc:194] Registered new tserver with Master: 7c4c210e830147cfb524aa90c4d319d4 (127.7.194.193:41967)
I20260619 02:34:30.454452 9061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:55729
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:34:30.462016 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.453756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:48247 (local address 127.7.194.254:44621)
0619 02:34:30.453860 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.453864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.453877 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:34:30.453910 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.453913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.453954 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.454035 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:30.454652 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.455487 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.456899 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.457114 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.457196 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.458606 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:30.458609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:30.458611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:30.458635 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:30.460223 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.460753 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.460756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.460757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.460830 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.461134 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.461136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.461137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.461315 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:30.461389 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.461764 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.461896 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:34:30.462037 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.452935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44621 (local address 127.7.194.193:48247)
0619 02:34:30.453235 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:30.453251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:30.453345 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:30.454088 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:30.454094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:30.454336 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:30.454506 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.454515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.455650 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.455664 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.456771 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.456784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.457249 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.457252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.457323 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.457883 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:30.457903 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:30.458437 (+ 534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:30.460362 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.460368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.460373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.460641 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.460928 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.460933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.460935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.461037 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.461417 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:30.461423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:30.461564 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 02:34:30.461694 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 02:34:30.461866 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:34:30.480877 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44621
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:30.606211 9274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:30.606557 9274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:30.606645 9274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:30.610930 9274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:30.611030 9274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:30.611104 9274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:34:30.611191 9274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:30.616755 9274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.194.254:44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:30.618052 9274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:30.619138 9274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:30.629119 9279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:30.629287 9274 server_base.cc:1034] running on GCE node
W20260619 02:34:30.629216 9280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:30.629119 9282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:30.630004 9274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:30.630694 9274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:30.631919 9274 hybrid_clock.cc:648] HybridClock initialized: now 1781836470631909 us; error 34 us; skew 500 ppm
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:30.634730 9274 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:34:30.635630 9274 webserver.cc:460] Webserver started at http://127.7.194.194:39441/ using document root <none> and password file <none>
I20260619 02:34:30.636224 9274 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:30.636323 9274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:30.636562 9274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:30.638731 9274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "3d960d24df6f4c469a3b09dc211bade8"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "1e3d71cbeda42f87d357bae9683bf23b"
server_key_iv: "b184ac9f18104b5618682ae2151117ea"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.639322 9274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "3d960d24df6f4c469a3b09dc211bade8"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "1e3d71cbeda42f87d357bae9683bf23b"
server_key_iv: "b184ac9f18104b5618682ae2151117ea"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.643572 9274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:30.646301 9289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:30.647676 9274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:34:30.647822 9274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "3d960d24df6f4c469a3b09dc211bade8"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "1e3d71cbeda42f87d357bae9683bf23b"
server_key_iv: "b184ac9f18104b5618682ae2151117ea"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.648000 9274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:30.660853 9274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:30.664111 9274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:30.664330 9274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:30.665879 9274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:30.665951 9274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:30.666013 9274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:30.666057 9274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:30.676954 9274 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:42557
I20260619 02:34:30.677018 9404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:42557 every 8 connection(s)
I20260619 02:34:30.678192 9274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:30.680716 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.666822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:36205 (local address 127.7.194.254:44621)
0619 02:34:30.667000 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.667007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.667876 (+ 869us) server_negotiation.cc:407] Connection header received
0619 02:34:30.669258 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.669264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.669347 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.669413 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:30.671391 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.671923 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.673844 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.674079 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.674163 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.677018 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:30.677023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:30.677026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:30.677055 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:30.678754 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.679347 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.679353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.679356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.679453 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.679752 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.679754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.679755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.679943 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:30.680021 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.680488 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.680594 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:34:30.681632 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.667226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44621 (local address 127.7.194.194:36205)
0619 02:34:30.667705 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:30.667767 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:30.668963 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:30.669903 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:30.669911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:30.670416 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:30.671222 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.671247 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.672068 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.672074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.673692 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.673704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.674272 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.674277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.674500 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.675261 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:30.675280 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:30.676830 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:30.678884 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.678891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.678907 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.679235 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.679554 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.679556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.679557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.679677 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.680031 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:30.680036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:30.680253 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 02:34:30.680429 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 02:34:30.680672 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":95,"threads_started":1}
I20260619 02:34:30.682732 9405 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44621
I20260619 02:34:30.682956 9405 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:30.683523 9405 heartbeater.cc:511] Master 127.7.194.254:44621 requested a full tablet report, sending...
I20260619 02:34:30.684615 9062 ts_manager.cc:194] Registered new tserver with Master: 3d960d24df6f4c469a3b09dc211bade8 (127.7.194.194:42557)
I20260619 02:34:30.685312 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:36205
I20260619 02:34:30.688411 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 9274
I20260619 02:34:30.688532 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:30.688814 7947 external_mini_cluster.cc:1428] Setting key 34175be1c78e05adf97d90c34211d811
I20260619 02:34:30.692503 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.685107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:36067 (local address 127.7.194.254:44621)
0619 02:34:30.685229 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.685235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.685440 (+ 205us) server_negotiation.cc:407] Connection header received
0619 02:34:30.685605 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.685608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.685656 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.685741 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:30.686211 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.686697 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.687651 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.687867 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.687948 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.689210 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:30.689214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:30.689223 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:30.689244 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:30.690814 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.691303 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.691305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.691306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.691375 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.691668 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.691669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.691671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.691869 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:30.691978 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.692240 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.692345 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:34:30.692518 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.685033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44621 (local address 127.7.194.194:36067)
0619 02:34:30.685352 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:30.685382 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:30.685488 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:30.685762 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:30.685765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:30.685972 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:30.686097 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.686104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.686831 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.686834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.687557 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.687565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.687974 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.687976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.688047 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.688645 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:30.688666 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:30.689064 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:30.690923 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.690929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.690934 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.691202 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.691468 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.691474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.691479 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.691580 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.691978 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:30.691982 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:30.692079 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 02:34:30.692190 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:34:30.692338 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":6,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:34:30.714680 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44621
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:30.831785 9414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:30.832087 9414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:30.832175 9414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:30.836385 9414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:30.836462 9414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:30.836560 9414 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:34:30.836634 9414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:30.842309 9414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.194.254:44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:30.843501 9414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:30.844599 9414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:30.855204 9422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:30.855233 9420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:30.855263 9419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:30.855701 9414 server_base.cc:1034] running on GCE node
I20260619 02:34:30.856164 9414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:30.856913 9414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:30.858103 9414 hybrid_clock.cc:648] HybridClock initialized: now 1781836470858077 us; error 43 us; skew 500 ppm
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:30.861042 9414 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:34:30.862490 9414 webserver.cc:460] Webserver started at http://127.7.194.195:33219/ using document root <none> and password file <none>
I20260619 02:34:30.863201 9414 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:30.863296 9414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:30.863543 9414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:30.865747 9414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "b2a364f738b64634bed7308c9504f93a"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "297859166bd36a7cb54168baaacc9fd5"
server_key_iv: "73d24d3caba6f705a5f297838f3305c8"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.866340 9414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "b2a364f738b64634bed7308c9504f93a"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "297859166bd36a7cb54168baaacc9fd5"
server_key_iv: "73d24d3caba6f705a5f297838f3305c8"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.870210 9414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:30.872685 9429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:30.874017 9414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:34:30.874150 9414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "b2a364f738b64634bed7308c9504f93a"
format_stamp: "Formatted at 2026-06-19 02:34:30 on dist-test-slave-9tkz"
server_key: "297859166bd36a7cb54168baaacc9fd5"
server_key_iv: "73d24d3caba6f705a5f297838f3305c8"
server_key_version: "encryptionkey@0"
I20260619 02:34:30.874262 9414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:30.901481 9414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:30.904784 9414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:30.905014 9414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:30.906819 9414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:30.906903 9414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:30.906967 9414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:30.907004 9414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:30.918177 9414 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:41137
I20260619 02:34:30.918459 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:41137 every 8 connection(s)
I20260619 02:34:30.919449 9414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:30.922065 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 9414
I20260619 02:34:30.922163 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:30.922498 7947 external_mini_cluster.cc:1428] Setting key 0352733c41f940569f6b429080e6b5ff
I20260619 02:34:30.922770 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.907579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:53351 (local address 127.7.194.254:44621)
0619 02:34:30.907711 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.907715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.908827 (+ 1112us) server_negotiation.cc:407] Connection header received
0619 02:34:30.910036 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.910040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.910108 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.910172 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:30.911861 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.912787 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.914316 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.914542 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.914660 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.918686 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:30.918692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:30.918695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:30.918734 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:30.920514 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.921036 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.921039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.921040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.921125 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.921474 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.921476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.921478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.921743 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:30.921870 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.922464 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.922605 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:34:30.923857 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.908192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44621 (local address 127.7.194.195:53351)
0619 02:34:30.908671 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:30.908715 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:30.909748 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:30.910644 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:30.910654 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:30.911062 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:30.911682 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.911707 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.912935 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.912939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.914187 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.914197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.914715 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.914718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.914909 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.916478 (+ 1569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:30.916502 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:30.918457 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:30.920604 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.920609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.920629 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.920894 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.921219 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.921221 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.921222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.921385 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.921872 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:30.921878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:30.922158 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 02:34:30.922365 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 02:34:30.922719 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":139,"threads_started":1}
I20260619 02:34:30.925159 9545 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44621
I20260619 02:34:30.925345 9545 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:30.925971 9545 heartbeater.cc:511] Master 127.7.194.254:44621 requested a full tablet report, sending...
I20260619 02:34:30.927035 9061 ts_manager.cc:194] Registered new tserver with Master: b2a364f738b64634bed7308c9504f93a (127.7.194.195:41137)
I20260619 02:34:30.927837 9061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:53351
I20260619 02:34:30.934459 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.927359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:42033 (local address 127.7.194.254:44621)
0619 02:34:30.927491 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.927496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.927664 (+ 168us) server_negotiation.cc:407] Connection header received
0619 02:34:30.927765 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.927767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.927804 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.927869 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:30.928389 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.928872 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.929796 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.930028 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.930136 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.931281 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:30.931285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:30.931288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:30.931313 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:30.932786 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.933232 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.933237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.933239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.933316 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.933631 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.933634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.933636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.933804 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:30.933927 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.934188 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.934296 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:34:30.934489 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.927291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44621 (local address 127.7.194.195:42033)
0619 02:34:30.927579 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:30.927595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:30.927689 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:30.927915 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:30.927920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:30.928152 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:30.928279 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.928286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.929023 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.929026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.929684 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.929692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.930137 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.930140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.930229 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.930793 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:30.930813 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:30.931145 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:30.932905 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.932910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.932912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.933132 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.933407 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:30.933411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:30.933413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:30.933499 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:30.933928 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:30.933931 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:30.934028 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 02:34:30.934136 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 02:34:30.934285 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
I20260619 02:34:30.936530 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:34:30 dist-test-slave-9tkz krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:30.963168 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.954531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.7.194.254:44621)
0619 02:34:30.954661 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.954664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.954676 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:34:30.954718 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.954720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.954758 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.954851 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:30.955577 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.956146 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.956991 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.957212 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.957325 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.959820 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:30.959824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:30.959826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:30.959852 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:30.961328 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.961862 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.961866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.961868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.961954 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:30.962240 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:30.962244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:30.962246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:30.962482 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:30.962618 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.962863 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.962969 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:34:30.966573 9061 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:34:30.969893 9061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:30.984962 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.979377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:42557 (local address 127.0.0.1:47144)
0619 02:34:30.979969 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:30.979982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:30.980063 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:30.980697 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:30.980702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:30.980718 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:30.980877 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.980885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.982376 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.982379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.983549 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.983558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.984416 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.984420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.984532 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.984570 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:34:30.984690 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 02:34:30.984811 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260619 02:34:30.984972 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.979572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.7.194.194:42557)
0619 02:34:30.980291 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.980301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.980320 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:34:30.980395 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.980399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.980556 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.981025 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:30.981161 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.982242 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.983667 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.984294 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.984416 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.984598 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.984718 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.984811 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":117,"threads_started":1}
I20260619 02:34:30.985255 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.978736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41967 (local address 127.0.0.1:39746)
0619 02:34:30.979789 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:30.979876 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:30.980040 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:30.980742 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:30.980746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:30.980761 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:30.980924 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.980929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.981932 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.981937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.983404 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.983415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.984683 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.984686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.984891 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.984945 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:34:30.985017 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:34:30.985101 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":133,"thread_start_us":79,"threads_started":1}
I20260619 02:34:30.985687 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.979132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.7.194.195:41137)
0619 02:34:30.979665 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.979676 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.980129 (+ 453us) server_negotiation.cc:407] Connection header received
0619 02:34:30.980211 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.980216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.980415 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.980567 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:30.980857 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.982054 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.983569 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.984381 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.984481 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.984631 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.985409 (+ 778us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.985514 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":97,"threads_started":1}
I20260619 02:34:30.985680 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.978884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.7.194.193:41967)
0619 02:34:30.979665 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:30.979674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:30.979966 (+ 292us) server_negotiation.cc:407] Connection header received
0619 02:34:30.980194 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:30.980200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:30.980421 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:30.980567 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:30.981031 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.981826 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.983569 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.984377 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.984481 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.984617 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:30.985442 (+ 825us) server_negotiation.cc:299] Negotiation successful
0619 02:34:30.985534 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":113,"threads_started":1}
I20260619 02:34:30.986578 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:30.978763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:41137 (local address 127.0.0.1:50504)
0619 02:34:30.979792 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:30.979841 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:30.979983 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:30.980572 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:30.980578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:30.980600 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:30.980736 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.980742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.982156 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.982160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.983440 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:30.983446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:30.984683 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:30.984686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:30.984886 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:30.984945 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 02:34:30.985017 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:34:30.985216 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":204,"thread_start_us":94,"threads_started":1}
I20260619 02:34:30.988155 9339 tablet_service.cc:1467] Processing CreateTablet for tablet b0d2bbd331da48d3ae06f9671d64f834 (DEFAULT_TABLE table=test-table [id=8a6f86b5a50a40e89e18921ec7bffed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:30.988435 9199 tablet_service.cc:1467] Processing CreateTablet for tablet b0d2bbd331da48d3ae06f9671d64f834 (DEFAULT_TABLE table=test-table [id=8a6f86b5a50a40e89e18921ec7bffed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:30.989073 9339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d2bbd331da48d3ae06f9671d64f834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:30.989430 9199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d2bbd331da48d3ae06f9671d64f834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:30.989082 9479 tablet_service.cc:1467] Processing CreateTablet for tablet b0d2bbd331da48d3ae06f9671d64f834 (DEFAULT_TABLE table=test-table [id=8a6f86b5a50a40e89e18921ec7bffed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:30.990026 9479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d2bbd331da48d3ae06f9671d64f834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:30.996366 9568 tablet_bootstrap.cc:492] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: Bootstrap starting.
I20260619 02:34:30.997731 9569 tablet_bootstrap.cc:492] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: Bootstrap starting.
I20260619 02:34:30.997862 9570 tablet_bootstrap.cc:492] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: Bootstrap starting.
I20260619 02:34:30.999014 9568 tablet_bootstrap.cc:654] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:30.999742 9569 tablet_bootstrap.cc:654] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:30.999871 9570 tablet_bootstrap.cc:654] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:30.999948 9568 log.cc:824] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:31.000556 9569 log.cc:824] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:31.000607 9570 log.cc:824] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:31.002033 9568 tablet_bootstrap.cc:492] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: No bootstrap required, opened a new log
I20260619 02:34:31.002246 9568 ts_tablet_manager.cc:1397] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:34:31.002321 9569 tablet_bootstrap.cc:492] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: No bootstrap required, opened a new log
I20260619 02:34:31.002494 9569 ts_tablet_manager.cc:1397] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:34:31.005519 9568 raft_consensus.cc:348] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } }
I20260619 02:34:31.005892 9568 raft_consensus.cc:374] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:31.005950 9568 raft_consensus.cc:729] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d960d24df6f4c469a3b09dc211bade8, State: Initialized, Role: FOLLOWER
I20260619 02:34:31.006409 9568 consensus_queue.cc:260] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } }
I20260619 02:34:31.006395 9569 raft_consensus.cc:348] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } }
I20260619 02:34:31.006635 9569 raft_consensus.cc:374] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:31.006681 9569 raft_consensus.cc:729] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4c210e830147cfb524aa90c4d319d4, State: Initialized, Role: FOLLOWER
I20260619 02:34:31.007158 9570 tablet_bootstrap.cc:492] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: No bootstrap required, opened a new log
I20260619 02:34:31.007236 9569 consensus_queue.cc:260] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } }
I20260619 02:34:31.007370 9570 ts_tablet_manager.cc:1397] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260619 02:34:31.007493 9568 ts_tablet_manager.cc:1428] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:34:31.007958 9265 heartbeater.cc:503] Master 127.7.194.254:44621 was elected leader, sending a full tablet report...
I20260619 02:34:31.008030 9405 heartbeater.cc:503] Master 127.7.194.254:44621 was elected leader, sending a full tablet report...
I20260619 02:34:31.008282 9569 ts_tablet_manager.cc:1428] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:34:31.011147 9570 raft_consensus.cc:348] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } }
I20260619 02:34:31.011615 9570 raft_consensus.cc:374] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:31.011703 9570 raft_consensus.cc:729] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a364f738b64634bed7308c9504f93a, State: Initialized, Role: FOLLOWER
I20260619 02:34:31.012226 9570 consensus_queue.cc:260] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } }
I20260619 02:34:31.013305 9570 ts_tablet_manager.cc:1428] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:34:31.013645 9545 heartbeater.cc:503] Master 127.7.194.254:44621 was elected leader, sending a full tablet report...
I20260619 02:34:31.072919 9574 raft_consensus.cc:482] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:31.073149 9574 raft_consensus.cc:504] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } }
I20260619 02:34:31.074378 9574 leader_election.cc:288] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c4c210e830147cfb524aa90c4d319d4 (127.7.194.193:41967), b2a364f738b64634bed7308c9504f93a (127.7.194.195:41137)
I20260619 02:34:31.078713 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.074973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:57905 (local address 127.7.194.193:41967)
0619 02:34:31.075119 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.075124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.075139 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:34:31.075196 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.075198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.075249 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.075318 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:31.075634 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.076183 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.077376 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.078033 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.078152 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.078206 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.078474 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.078569 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:34:31.078712 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.074723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41967 (local address 127.7.194.194:57905)
0619 02:34:31.074875 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:31.074894 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:31.075008 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:31.075369 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:31.075372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:31.075385 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:31.075516 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.075521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.076311 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.076315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.077230 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.077239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.078185 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.078188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.078325 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.078345 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 02:34:31.078472 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 02:34:31.078562 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":48}
I20260619 02:34:31.079519 9219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2bbd331da48d3ae06f9671d64f834" candidate_uuid: "3d960d24df6f4c469a3b09dc211bade8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c210e830147cfb524aa90c4d319d4" is_pre_election: true
I20260619 02:34:31.079644 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.074723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:41137 (local address 127.7.194.194:46553)
0619 02:34:31.075250 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:31.075259 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:31.075380 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:31.075808 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:31.075811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:31.075828 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:31.076339 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.076355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.077089 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.077096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.078310 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.078318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.079211 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.079217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.079314 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.079348 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 02:34:31.079421 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:34:31.079482 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":167,"thread_start_us":156,"threads_started":1}
I20260619 02:34:31.079701 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.074841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:46553 (local address 127.7.194.195:41137)
0619 02:34:31.075001 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.075005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.075603 (+ 598us) server_negotiation.cc:407] Connection header received
0619 02:34:31.075649 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.075670 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.075709 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.075799 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:31.076472 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.076969 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.078438 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.078982 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.079094 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.079141 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.079464 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.079566 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 02:34:31.079908 9219 raft_consensus.cc:2454] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d960d24df6f4c469a3b09dc211bade8 in term 0.
I20260619 02:34:31.080331 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2bbd331da48d3ae06f9671d64f834" candidate_uuid: "3d960d24df6f4c469a3b09dc211bade8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a364f738b64634bed7308c9504f93a" is_pre_election: true
I20260619 02:34:31.080446 9291 leader_election.cc:302] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d960d24df6f4c469a3b09dc211bade8, 7c4c210e830147cfb524aa90c4d319d4; no voters:
I20260619 02:34:31.080667 9499 raft_consensus.cc:2454] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d960d24df6f4c469a3b09dc211bade8 in term 0.
I20260619 02:34:31.080797 9574 raft_consensus.cc:2781] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:31.080891 9574 raft_consensus.cc:482] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:31.080952 9574 raft_consensus.cc:3037] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:31.082130 9574 raft_consensus.cc:504] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } }
I20260619 02:34:31.082501 9574 leader_election.cc:288] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [CANDIDATE]: Term 1 election: Requested vote from peers 7c4c210e830147cfb524aa90c4d319d4 (127.7.194.193:41967), b2a364f738b64634bed7308c9504f93a (127.7.194.195:41137)
I20260619 02:34:31.082896 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2bbd331da48d3ae06f9671d64f834" candidate_uuid: "3d960d24df6f4c469a3b09dc211bade8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a364f738b64634bed7308c9504f93a"
I20260619 02:34:31.082988 9219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2bbd331da48d3ae06f9671d64f834" candidate_uuid: "3d960d24df6f4c469a3b09dc211bade8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c210e830147cfb524aa90c4d319d4"
I20260619 02:34:31.083029 9499 raft_consensus.cc:3037] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:31.083170 9219 raft_consensus.cc:3037] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:31.084276 9499 raft_consensus.cc:2454] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d960d24df6f4c469a3b09dc211bade8 in term 1.
I20260619 02:34:31.084335 9219 raft_consensus.cc:2454] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d960d24df6f4c469a3b09dc211bade8 in term 1.
I20260619 02:34:31.084600 9293 leader_election.cc:302] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d960d24df6f4c469a3b09dc211bade8, b2a364f738b64634bed7308c9504f93a; no voters:
I20260619 02:34:31.084791 9574 raft_consensus.cc:2781] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:31.084988 9574 raft_consensus.cc:686] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 1 LEADER]: Becoming Leader. State: Replica: 3d960d24df6f4c469a3b09dc211bade8, State: Running, Role: LEADER
I20260619 02:34:31.085350 9574 consensus_queue.cc:237] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } }
I20260619 02:34:31.088778 9061 catalog_manager.cc:5495] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d960d24df6f4c469a3b09dc211bade8 (127.7.194.194). New cstate: current_term: 1 leader_uuid: "3d960d24df6f4c469a3b09dc211bade8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } health_report { overall_health: HEALTHY } } }
I20260619 02:34:31.117961 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.113741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.7.194.194:42557)
0619 02:34:31.113891 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.113896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.114008 (+ 112us) server_negotiation.cc:407] Connection header received
0619 02:34:31.114274 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.114276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.114334 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.114453 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:31.114721 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.115253 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.116244 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.116495 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.116633 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.117023 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:31.117469 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:31.117619 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.117681 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.117748 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:34:31.124125 9219 raft_consensus.cc:1261] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Refusing update from remote peer 3d960d24df6f4c469a3b09dc211bade8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:31.124322 9499 raft_consensus.cc:1261] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [term 1 FOLLOWER]: Refusing update from remote peer 3d960d24df6f4c469a3b09dc211bade8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:31.124958 9578 consensus_queue.cc:1035] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:31.125129 9574 consensus_queue.cc:1035] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:31.135685 9582 mvcc.cc:205] Tried to move back new op lower bound from 7298402185716277248 to 7298402185568346112. Current Snapshot: MvccSnapshot[applied={T|T < 7298402185716277248}]
I20260619 02:34:31.144717 9062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:31.144901 9062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:31.147606 9062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685: Sending DeleteTablet for 3 replicas of tablet b0d2bbd331da48d3ae06f9671d64f834
I20260619 02:34:31.148412 9339 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d2bbd331da48d3ae06f9671d64f834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:31 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:47144
I20260619 02:34:31.148411 9199 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d2bbd331da48d3ae06f9671d64f834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:31 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:39746
I20260619 02:34:31.148416 9479 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d2bbd331da48d3ae06f9671d64f834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:31 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:50504
I20260619 02:34:31.148918 9591 tablet_replica.cc:330] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: stopping tablet replica
I20260619 02:34:31.149248 9591 raft_consensus.cc:2229] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:34:31.149709 9591 raft_consensus.cc:2258] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:31.149771 9592 tablet_replica.cc:330] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: stopping tablet replica
I20260619 02:34:31.149945 9062 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:34:31.150174 9592 raft_consensus.cc:2229] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 02:34:31.150383 9062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:31.150524 9592 raft_consensus.cc:2258] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:31.151351 9591 ts_tablet_manager.cc:1905] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:31.152992 9592 ts_tablet_manager.cc:1905] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:31.154392 9591 ts_tablet_manager.cc:1918] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:31.154492 9591 log.cc:1196] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/b0d2bbd331da48d3ae06f9671d64f834
I20260619 02:34:31.154811 9591 ts_tablet_manager.cc:1939] T b0d2bbd331da48d3ae06f9671d64f834 P 3d960d24df6f4c469a3b09dc211bade8: Deleting consensus metadata
I20260619 02:34:31.156038 9046 catalog_manager.cc:4841] TS 3d960d24df6f4c469a3b09dc211bade8 (127.7.194.194:42557): tablet b0d2bbd331da48d3ae06f9671d64f834 (table test-table [id=8a6f86b5a50a40e89e18921ec7bffed9]) successfully deleted
I20260619 02:34:31.156569 9592 ts_tablet_manager.cc:1918] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:31.156637 9592 log.cc:1196] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/b0d2bbd331da48d3ae06f9671d64f834
I20260619 02:34:31.156704 9479 tablet_service.cc:1467] Processing CreateTablet for tablet dd6f11e3b46149d2bd4dd84bfd0166a9 (DEFAULT_TABLE table=test-table [id=445c1dd26ac847e693edc55d4f062543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:31.156843 9592 ts_tablet_manager.cc:1939] T b0d2bbd331da48d3ae06f9671d64f834 P 7c4c210e830147cfb524aa90c4d319d4: Deleting consensus metadata
I20260619 02:34:31.156996 9479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6f11e3b46149d2bd4dd84bfd0166a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:31.157972 9339 tablet_service.cc:1467] Processing CreateTablet for tablet dd6f11e3b46149d2bd4dd84bfd0166a9 (DEFAULT_TABLE table=test-table [id=445c1dd26ac847e693edc55d4f062543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:31.158164 9339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6f11e3b46149d2bd4dd84bfd0166a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:31.159319 9570 tablet_bootstrap.cc:492] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: Bootstrap starting.
I20260619 02:34:31.159947 9568 tablet_bootstrap.cc:492] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: Bootstrap starting.
I20260619 02:34:31.160319 9047 catalog_manager.cc:4841] TS 7c4c210e830147cfb524aa90c4d319d4 (127.7.194.193:41967): tablet b0d2bbd331da48d3ae06f9671d64f834 (table test-table [id=8a6f86b5a50a40e89e18921ec7bffed9]) successfully deleted
I20260619 02:34:31.160387 9570 tablet_bootstrap.cc:654] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:31.161156 9199 tablet_service.cc:1467] Processing CreateTablet for tablet dd6f11e3b46149d2bd4dd84bfd0166a9 (DEFAULT_TABLE table=test-table [id=445c1dd26ac847e693edc55d4f062543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:31.161330 9568 tablet_bootstrap.cc:654] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:31.161415 9199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6f11e3b46149d2bd4dd84bfd0166a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:31.161634 9593 tablet_replica.cc:330] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: stopping tablet replica
I20260619 02:34:31.162014 9593 raft_consensus.cc:2229] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:31.162381 9593 raft_consensus.cc:2258] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:31.162550 9568 tablet_bootstrap.cc:492] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: No bootstrap required, opened a new log
I20260619 02:34:31.162639 9568 ts_tablet_manager.cc:1397] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 02:34:31.163107 9568 raft_consensus.cc:348] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.163241 9568 raft_consensus.cc:374] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:31.163282 9568 raft_consensus.cc:729] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d960d24df6f4c469a3b09dc211bade8, State: Initialized, Role: FOLLOWER
I20260619 02:34:31.163439 9568 consensus_queue.cc:260] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.163674 9568 ts_tablet_manager.cc:1428] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:34:31.164108 9569 tablet_bootstrap.cc:492] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: Bootstrap starting.
I20260619 02:34:31.164108 9593 ts_tablet_manager.cc:1905] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:31.165068 9569 tablet_bootstrap.cc:654] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:31.166483 9593 ts_tablet_manager.cc:1918] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:31.166599 9593 log.cc:1196] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/b0d2bbd331da48d3ae06f9671d64f834
I20260619 02:34:31.166946 9593 ts_tablet_manager.cc:1939] T b0d2bbd331da48d3ae06f9671d64f834 P b2a364f738b64634bed7308c9504f93a: Deleting consensus metadata
I20260619 02:34:31.167580 9569 tablet_bootstrap.cc:492] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: No bootstrap required, opened a new log
I20260619 02:34:31.167652 9569 ts_tablet_manager.cc:1397] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260619 02:34:31.168151 9049 catalog_manager.cc:4841] TS b2a364f738b64634bed7308c9504f93a (127.7.194.195:41137): tablet b0d2bbd331da48d3ae06f9671d64f834 (table test-table [id=8a6f86b5a50a40e89e18921ec7bffed9]) successfully deleted
I20260619 02:34:31.168381 9569 raft_consensus.cc:348] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.168498 9569 raft_consensus.cc:374] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:31.168535 9569 raft_consensus.cc:729] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4c210e830147cfb524aa90c4d319d4, State: Initialized, Role: FOLLOWER
I20260619 02:34:31.168684 9569 consensus_queue.cc:260] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.169287 9570 tablet_bootstrap.cc:492] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: No bootstrap required, opened a new log
I20260619 02:34:31.169366 9570 ts_tablet_manager.cc:1397] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260619 02:34:31.169811 9570 raft_consensus.cc:348] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.169910 9570 raft_consensus.cc:374] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:31.169939 9570 raft_consensus.cc:729] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a364f738b64634bed7308c9504f93a, State: Initialized, Role: FOLLOWER
I20260619 02:34:31.170047 9570 consensus_queue.cc:260] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.170308 9570 ts_tablet_manager.cc:1428] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260619 02:34:31.170733 9546 tablet.cc:2369] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:34:31.171648 9569 ts_tablet_manager.cc:1428] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20260619 02:34:31.179379 9406 tablet.cc:2369] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:34:31.196025 9266 tablet.cc:2369] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:34:31.638446 9576 raft_consensus.cc:482] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:31.638617 9576 raft_consensus.cc:504] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.639703 9576 leader_election.cc:288] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c4c210e830147cfb524aa90c4d319d4 (127.7.194.193:41967), 3d960d24df6f4c469a3b09dc211bade8 (127.7.194.194:42557)
I20260619 02:34:31.643702 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.640077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:40115 (local address 127.7.194.194:42557)
0619 02:34:31.640278 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.640284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.640306 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:34:31.640450 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.640453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.640498 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.640572 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:31.640949 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.641522 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.642679 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.643094 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.643225 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.643280 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.643472 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.643551 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
I20260619 02:34:31.643715 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.639944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:42557 (local address 127.7.194.195:40115)
0619 02:34:31.640209 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:31.640225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:31.640334 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:31.640643 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:31.640648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:31.640667 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:31.640826 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.640832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.641700 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.641708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.642554 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.642563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.643218 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.643222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.643349 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.643368 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:34:31.643458 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:34:31.643540 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":33,"thread_start_us":107,"threads_started":1}
I20260619 02:34:31.643720 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.640013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:42031 (local address 127.7.194.193:41967)
0619 02:34:31.640321 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.640326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.640585 (+ 259us) server_negotiation.cc:407] Connection header received
0619 02:34:31.640702 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.640704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.640743 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.640805 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:31.641071 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.641645 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.642677 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.643101 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.643236 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.643291 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.643504 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.643584 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":76,"threads_started":1}
I20260619 02:34:31.643719 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.639946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41967 (local address 127.7.194.195:42031)
0619 02:34:31.640503 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:31.640513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:31.640616 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:31.640837 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:31.640840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:31.640858 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:31.640965 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.640970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.641750 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.641755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.642558 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.642566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.643223 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.643227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.643325 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.643400 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 02:34:31.643486 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:34:31.643565 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":311,"thread_start_us":82,"threads_started":1}
I20260619 02:34:31.644364 9219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6f11e3b46149d2bd4dd84bfd0166a9" candidate_uuid: "b2a364f738b64634bed7308c9504f93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c210e830147cfb524aa90c4d319d4" is_pre_election: true
I20260619 02:34:31.644419 9359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6f11e3b46149d2bd4dd84bfd0166a9" candidate_uuid: "b2a364f738b64634bed7308c9504f93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d960d24df6f4c469a3b09dc211bade8" is_pre_election: true
I20260619 02:34:31.644536 9219 raft_consensus.cc:2454] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a364f738b64634bed7308c9504f93a in term 0.
I20260619 02:34:31.644675 9359 raft_consensus.cc:2454] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a364f738b64634bed7308c9504f93a in term 0.
I20260619 02:34:31.644956 9431 leader_election.cc:302] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c4c210e830147cfb524aa90c4d319d4, b2a364f738b64634bed7308c9504f93a; no voters:
I20260619 02:34:31.645205 9576 raft_consensus.cc:2781] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:31.645248 9576 raft_consensus.cc:482] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:31.645282 9576 raft_consensus.cc:3037] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:31.646360 9576 raft_consensus.cc:504] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.646772 9576 leader_election.cc:288] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [CANDIDATE]: Term 1 election: Requested vote from peers 7c4c210e830147cfb524aa90c4d319d4 (127.7.194.193:41967), 3d960d24df6f4c469a3b09dc211bade8 (127.7.194.194:42557)
I20260619 02:34:31.647117 9219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6f11e3b46149d2bd4dd84bfd0166a9" candidate_uuid: "b2a364f738b64634bed7308c9504f93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c210e830147cfb524aa90c4d319d4"
I20260619 02:34:31.647203 9359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6f11e3b46149d2bd4dd84bfd0166a9" candidate_uuid: "b2a364f738b64634bed7308c9504f93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d960d24df6f4c469a3b09dc211bade8"
I20260619 02:34:31.647281 9219 raft_consensus.cc:3037] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:31.647324 9359 raft_consensus.cc:3037] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:31.648339 9219 raft_consensus.cc:2454] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a364f738b64634bed7308c9504f93a in term 1.
I20260619 02:34:31.648342 9359 raft_consensus.cc:2454] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a364f738b64634bed7308c9504f93a in term 1.
I20260619 02:34:31.648620 9431 leader_election.cc:302] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c4c210e830147cfb524aa90c4d319d4, b2a364f738b64634bed7308c9504f93a; no voters:
I20260619 02:34:31.648785 9576 raft_consensus.cc:2781] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:31.648986 9576 raft_consensus.cc:686] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 1 LEADER]: Becoming Leader. State: Replica: b2a364f738b64634bed7308c9504f93a, State: Running, Role: LEADER
I20260619 02:34:31.649250 9576 consensus_queue.cc:237] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.651662 9061 catalog_manager.cc:5495] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a364f738b64634bed7308c9504f93a (127.7.194.195). New cstate: current_term: 1 leader_uuid: "b2a364f738b64634bed7308c9504f93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } health_report { overall_health: HEALTHY } } }
Jun 19 02:34:31 dist-test-slave-9tkz krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:31.753365 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.743537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44621 (local address 127.0.0.1:47608)
0619 02:34:31.743977 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:31.743989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:31.744098 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:31.744462 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:31.744465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:31.744938 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:31.745045 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.745051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.745924 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.745927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.746487 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.746494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.746932 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.746934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.746993 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.747700 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:31.747720 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:31.749501 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:31.751294 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:31.751304 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:31.751311 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:31.751571 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:31.751900 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:31.751903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:31.751905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:31.751981 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:31.752493 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:31.752503 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:31.752669 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 02:34:31.752808 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 02:34:31.753045 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":133,"threads_started":1}
I20260619 02:34:31.753400 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.743577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.7.194.254:44621)
0619 02:34:31.743870 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.743879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.744070 (+ 191us) server_negotiation.cc:407] Connection header received
0619 02:34:31.744240 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.744242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.744287 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.744379 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:31.745176 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.745825 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.746606 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.746825 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.746914 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.749721 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:31.749725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:31.749727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:31.749747 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:31.751172 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:31.751692 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:31.751699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:31.751703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:31.751783 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:31.752086 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:31.752089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:31.752091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:31.752345 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:31.752451 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.752907 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.753148 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":217,"thread_start_us":112,"threads_started":1}
I20260619 02:34:31.757776 9061 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:47608:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:34:31.758548 9061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:31.763386 9339 tablet_service.cc:1467] Processing CreateTablet for tablet dc59fad266994b1abcf0d0ffc3fe638e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=60db0f01a3254a16ae45c76fb6309454]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:34:31.763374 9199 tablet_service.cc:1467] Processing CreateTablet for tablet dc59fad266994b1abcf0d0ffc3fe638e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=60db0f01a3254a16ae45c76fb6309454]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:34:31.763510 9479 tablet_service.cc:1467] Processing CreateTablet for tablet dc59fad266994b1abcf0d0ffc3fe638e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=60db0f01a3254a16ae45c76fb6309454]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:34:31.763785 9339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc59fad266994b1abcf0d0ffc3fe638e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:31.763787 9199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc59fad266994b1abcf0d0ffc3fe638e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:31.763827 9479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc59fad266994b1abcf0d0ffc3fe638e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:31.766273 9610 tablet_bootstrap.cc:492] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: Bootstrap starting.
I20260619 02:34:31.767266 9611 tablet_bootstrap.cc:492] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: Bootstrap starting.
I20260619 02:34:31.767342 9610 tablet_bootstrap.cc:654] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:31.767895 9612 tablet_bootstrap.cc:492] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: Bootstrap starting.
I20260619 02:34:31.768153 9611 tablet_bootstrap.cc:654] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:31.768486 9610 tablet_bootstrap.cc:492] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: No bootstrap required, opened a new log
I20260619 02:34:31.768573 9610 ts_tablet_manager.cc:1397] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:31.768860 9612 tablet_bootstrap.cc:654] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:31.769200 9610 raft_consensus.cc:348] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.769330 9610 raft_consensus.cc:374] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:31.769387 9610 raft_consensus.cc:729] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d960d24df6f4c469a3b09dc211bade8, State: Initialized, Role: FOLLOWER
I20260619 02:34:31.769586 9611 tablet_bootstrap.cc:492] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: No bootstrap required, opened a new log
I20260619 02:34:31.769536 9610 consensus_queue.cc:260] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.769776 9611 ts_tablet_manager.cc:1397] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:34:31.770037 9610 ts_tablet_manager.cc:1428] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 02:34:31.770131 9612 tablet_bootstrap.cc:492] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: No bootstrap required, opened a new log
I20260619 02:34:31.770222 9612 ts_tablet_manager.cc:1397] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:34:31.770259 9611 raft_consensus.cc:348] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.770381 9611 raft_consensus.cc:374] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:31.770424 9611 raft_consensus.cc:729] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4c210e830147cfb524aa90c4d319d4, State: Initialized, Role: FOLLOWER
I20260619 02:34:31.770617 9611 consensus_queue.cc:260] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.770718 9612 raft_consensus.cc:348] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.770850 9612 raft_consensus.cc:374] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:31.770802 9616 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.770884 9612 raft_consensus.cc:729] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a364f738b64634bed7308c9504f93a, State: Initialized, Role: FOLLOWER
I20260619 02:34:31.770944 9616 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:31.771065 9611 ts_tablet_manager.cc:1428] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:31.771015 9612 consensus_queue.cc:260] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.771270 9612 ts_tablet_manager.cc:1428] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:34:31.771258 9576 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.771414 9576 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:31.772236 9616 raft_consensus.cc:482] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:31.772346 9616 raft_consensus.cc:504] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.772696 9616 leader_election.cc:288] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c4c210e830147cfb524aa90c4d319d4 (127.7.194.193:41967), b2a364f738b64634bed7308c9504f93a (127.7.194.195:41137)
I20260619 02:34:31.773173 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc59fad266994b1abcf0d0ffc3fe638e" candidate_uuid: "3d960d24df6f4c469a3b09dc211bade8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a364f738b64634bed7308c9504f93a" is_pre_election: true
I20260619 02:34:31.773303 9499 raft_consensus.cc:2454] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d960d24df6f4c469a3b09dc211bade8 in term 0.
I20260619 02:34:31.773264 9219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc59fad266994b1abcf0d0ffc3fe638e" candidate_uuid: "3d960d24df6f4c469a3b09dc211bade8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c210e830147cfb524aa90c4d319d4" is_pre_election: true
I20260619 02:34:31.773399 9219 raft_consensus.cc:2454] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d960d24df6f4c469a3b09dc211bade8 in term 0.
I20260619 02:34:31.773643 9293 leader_election.cc:302] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d960d24df6f4c469a3b09dc211bade8, b2a364f738b64634bed7308c9504f93a; no voters:
I20260619 02:34:31.773801 9616 raft_consensus.cc:2781] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:31.773861 9616 raft_consensus.cc:482] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:31.773901 9616 raft_consensus.cc:3037] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:31.773253 9617 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.774042 9617 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:31.775174 9616 raft_consensus.cc:504] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.775563 9616 leader_election.cc:288] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [CANDIDATE]: Term 1 election: Requested vote from peers 7c4c210e830147cfb524aa90c4d319d4 (127.7.194.193:41967), b2a364f738b64634bed7308c9504f93a (127.7.194.195:41137)
I20260619 02:34:31.775846 9219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc59fad266994b1abcf0d0ffc3fe638e" candidate_uuid: "3d960d24df6f4c469a3b09dc211bade8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c210e830147cfb524aa90c4d319d4"
I20260619 02:34:31.775985 9219 raft_consensus.cc:3037] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:31.777005 9219 raft_consensus.cc:2454] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d960d24df6f4c469a3b09dc211bade8 in term 1.
I20260619 02:34:31.777297 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc59fad266994b1abcf0d0ffc3fe638e" candidate_uuid: "3d960d24df6f4c469a3b09dc211bade8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a364f738b64634bed7308c9504f93a"
I20260619 02:34:31.777338 9291 leader_election.cc:302] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d960d24df6f4c469a3b09dc211bade8, 7c4c210e830147cfb524aa90c4d319d4; no voters:
I20260619 02:34:31.777421 9499 raft_consensus.cc:3037] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:31.777503 9616 raft_consensus.cc:2781] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:31.777730 9616 raft_consensus.cc:686] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [term 1 LEADER]: Becoming Leader. State: Replica: 3d960d24df6f4c469a3b09dc211bade8, State: Running, Role: LEADER
I20260619 02:34:31.777834 9616 consensus_queue.cc:237] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } }
I20260619 02:34:31.778515 9618 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: TxnStatusTablet state changed. Reason: New leader 3d960d24df6f4c469a3b09dc211bade8. Latest consensus state: current_term: 1 leader_uuid: "3d960d24df6f4c469a3b09dc211bade8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.778592 9499 raft_consensus.cc:2454] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d960d24df6f4c469a3b09dc211bade8 in term 1.
I20260619 02:34:31.778630 9618 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:34:31.778892 9620 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:34:31.779053 9620 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:34:31.779413 9062 catalog_manager.cc:5495] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d960d24df6f4c469a3b09dc211bade8 (127.7.194.194). New cstate: current_term: 1 leader_uuid: "3d960d24df6f4c469a3b09dc211bade8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } health_report { overall_health: UNKNOWN } } }
I20260619 02:34:31.795282 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.791253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.7.194.194:42557)
0619 02:34:31.791377 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.791387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.791448 (+ 61us) server_negotiation.cc:407] Connection header received
0619 02:34:31.791627 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.791630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.791673 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.791786 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:31.792079 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.792660 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.794120 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.794348 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.794514 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.794810 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:31.794900 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:31.794979 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.795084 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.795139 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:34:31.795282 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.791155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:42557 (local address 127.0.0.1:47170)
0619 02:34:31.791389 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:31.791404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:31.791514 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:31.791823 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:31.791826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:31.791835 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:34:31.791965 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.791970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.792785 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.792789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.793987 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.793997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.794508 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.794512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.794613 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.794668 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:34:31.794994 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:34:31.795011 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:34:31.795074 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:34:31.795159 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260619 02:34:31.798242 9499 raft_consensus.cc:1261] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a [term 1 FOLLOWER]: Refusing update from remote peer 3d960d24df6f4c469a3b09dc211bade8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:31.798257 9219 raft_consensus.cc:1261] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Refusing update from remote peer 3d960d24df6f4c469a3b09dc211bade8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:31.798691 9616 consensus_queue.cc:1035] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:31.798924 9618 consensus_queue.cc:1035] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:31.800390 9617 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: TxnStatusTablet state changed. Reason: New leader 3d960d24df6f4c469a3b09dc211bade8. Latest consensus state: current_term: 1 leader_uuid: "3d960d24df6f4c469a3b09dc211bade8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.800501 9617 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:31.800657 9576 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: TxnStatusTablet state changed. Reason: New leader 3d960d24df6f4c469a3b09dc211bade8. Latest consensus state: current_term: 1 leader_uuid: "3d960d24df6f4c469a3b09dc211bade8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.800761 9576 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:31.802847 9618 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d960d24df6f4c469a3b09dc211bade8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.802968 9618 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:34:31.803263 9617 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d960d24df6f4c469a3b09dc211bade8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.803356 9617 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P 7c4c210e830147cfb524aa90c4d319d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:31.803606 9616 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d960d24df6f4c469a3b09dc211bade8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.803705 9616 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P 3d960d24df6f4c469a3b09dc211bade8: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:34:31.804208 9624 mvcc.cc:205] Tried to move back new op lower bound from 7298402188481200128 to 7298402188403441664. Current Snapshot: MvccSnapshot[applied={T|T < 7298402188481200128}]
I20260619 02:34:31.804354 9576 tablet_replica.cc:439] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d960d24df6f4c469a3b09dc211bade8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 } } peers { permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 } } peers { permanent_uuid: "b2a364f738b64634bed7308c9504f93a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41137 } } }
I20260619 02:34:31.804486 9576 tablet_replica.cc:442] T dc59fad266994b1abcf0d0ffc3fe638e P b2a364f738b64634bed7308c9504f93a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:31.806680 9623 mvcc.cc:205] Tried to move back new op lower bound from 7298402188481200128 to 7298402188403441664. Current Snapshot: MvccSnapshot[applied={T|T < 7298402188481200128}]
I20260619 02:34:31.816920 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.812753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.7.194.195:41137)
0619 02:34:31.813001 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.813014 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.813028 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:31.813188 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.813190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.813241 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.813341 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:31.813632 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.814159 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.815339 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.815565 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.815638 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.815929 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:31.816444 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:31.816545 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.816637 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.816719 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":142,"threads_started":1}
I20260619 02:34:31.827898 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.824256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:33649 (local address 127.7.194.194:42557)
0619 02:34:31.824384 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.824389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.824541 (+ 152us) server_negotiation.cc:407] Connection header received
0619 02:34:31.824696 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.824699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.824753 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.824831 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:31.825091 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.825765 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.826789 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.827015 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.827127 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.827406 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:31.827486 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:31.827564 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.827684 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.827737 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:34:31.827903 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.824143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:42557 (local address 127.7.194.195:33649)
0619 02:34:31.824456 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:31.824470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:31.824594 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:31.824846 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:31.824848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:31.824854 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:34:31.824955 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.824960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.825868 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.825874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.826661 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.826670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.827127 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.827131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.827227 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.827268 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:34:31.827577 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:34:31.827585 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:34:31.827672 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:34:31.827751 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":118,"threads_started":1}
I20260619 02:34:31.835260 9219 raft_consensus.cc:1261] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Refusing update from remote peer b2a364f738b64634bed7308c9504f93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:31.835336 9359 raft_consensus.cc:1261] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [term 1 FOLLOWER]: Refusing update from remote peer b2a364f738b64634bed7308c9504f93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:31.835736 9601 consensus_queue.cc:1035] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4c210e830147cfb524aa90c4d319d4" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:31.836233 9601 consensus_queue.cc:1035] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d960d24df6f4c469a3b09dc211bade8" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:31.858922 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.855103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:41137 (local address 127.7.194.194:51897)
0619 02:34:31.855440 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:31.855457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:31.855557 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:31.855817 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:31.855820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:31.855828 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:34:31.855941 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.855947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.856839 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.856841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.857623 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:31.857634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.858197 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.858199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:31.858255 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.858294 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:34:31.858609 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:34:31.858617 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:34:31.858708 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 02:34:31.858789 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
I20260619 02:34:31.858922 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.855201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:51897 (local address 127.7.194.195:41137)
0619 02:34:31.855434 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.855439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.855508 (+ 69us) server_negotiation.cc:407] Connection header received
0619 02:34:31.855665 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.855668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.855717 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.855806 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:31.856051 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.856710 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.857755 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.858048 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.858108 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.858448 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:31.858525 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:31.858605 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.858720 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.858771 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260619 02:34:31.882086 9061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:31.882282 9061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:31.884765 9061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c5e3c439ce144529ccb003785afe685: Sending DeleteTablet for 3 replicas of tablet dd6f11e3b46149d2bd4dd84bfd0166a9
I20260619 02:34:31.885397 9199 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6f11e3b46149d2bd4dd84bfd0166a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:31 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:39746
I20260619 02:34:31.885633 9339 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6f11e3b46149d2bd4dd84bfd0166a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:31 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:47144
I20260619 02:34:31.885908 9479 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6f11e3b46149d2bd4dd84bfd0166a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:31 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:50504
I20260619 02:34:31.886425 9645 tablet_replica.cc:330] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: stopping tablet replica
I20260619 02:34:31.886605 9645 raft_consensus.cc:2229] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:31.886773 9645 raft_consensus.cc:2258] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:31.887677 9645 ts_tablet_manager.cc:1905] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:31.889772 9647 tablet_replica.cc:330] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: stopping tablet replica
I20260619 02:34:31.889925 9647 raft_consensus.cc:2229] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:34:31.890264 9647 raft_consensus.cc:2258] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:31.890555 9646 tablet_replica.cc:330] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: stopping tablet replica
I20260619 02:34:31.890708 9646 raft_consensus.cc:2229] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:31.890836 9646 raft_consensus.cc:2258] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:31.891010 9647 ts_tablet_manager.cc:1905] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:31.891283 9645 ts_tablet_manager.cc:1918] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:34:31.891408 9645 log.cc:1196] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/dd6f11e3b46149d2bd4dd84bfd0166a9
I20260619 02:34:31.891506 9646 ts_tablet_manager.cc:1905] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:31.891834 9645 ts_tablet_manager.cc:1939] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 7c4c210e830147cfb524aa90c4d319d4: Deleting consensus metadata
I20260619 02:34:31.892629 9047 catalog_manager.cc:4841] TS 7c4c210e830147cfb524aa90c4d319d4 (127.7.194.193:41967): tablet dd6f11e3b46149d2bd4dd84bfd0166a9 (table test-table [id=445c1dd26ac847e693edc55d4f062543]) successfully deleted
I20260619 02:34:31.894394 9646 ts_tablet_manager.cc:1918] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:34:31.894474 9646 log.cc:1196] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/dd6f11e3b46149d2bd4dd84bfd0166a9
I20260619 02:34:31.894729 9646 ts_tablet_manager.cc:1939] T dd6f11e3b46149d2bd4dd84bfd0166a9 P 3d960d24df6f4c469a3b09dc211bade8: Deleting consensus metadata
I20260619 02:34:31.895169 9647 ts_tablet_manager.cc:1918] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:34:31.895247 9647 log.cc:1196] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/dd6f11e3b46149d2bd4dd84bfd0166a9
I20260619 02:34:31.895448 9046 catalog_manager.cc:4841] TS 3d960d24df6f4c469a3b09dc211bade8 (127.7.194.194:42557): tablet dd6f11e3b46149d2bd4dd84bfd0166a9 (table test-table [id=445c1dd26ac847e693edc55d4f062543]) successfully deleted
I20260619 02:34:31.895552 9647 ts_tablet_manager.cc:1939] T dd6f11e3b46149d2bd4dd84bfd0166a9 P b2a364f738b64634bed7308c9504f93a: Deleting consensus metadata
I20260619 02:34:31.896310 9049 catalog_manager.cc:4841] TS b2a364f738b64634bed7308c9504f93a (127.7.194.195:41137): tablet dd6f11e3b46149d2bd4dd84bfd0166a9 (table test-table [id=445c1dd26ac847e693edc55d4f062543]) successfully deleted
Jun 19 02:34:31 dist-test-slave-9tkz krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836470, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:31.908058 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.887448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.7.194.193:41967)
0619 02:34:31.887563 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.887568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.895791 (+ 8223us) server_negotiation.cc:407] Connection header received
0619 02:34:31.895929 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.895932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.895983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.896069 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:31.896940 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.897735 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.898915 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.899241 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.899324 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.903516 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:31.903531 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:31.903552 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:31.903584 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:31.905715 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:31.906268 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:31.906276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:31.906277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:31.906380 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:31.906708 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:31.906711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:31.906713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:31.907076 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:31.907279 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.907704 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.907867 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 02:34:31.909446 9259 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39758
I20260619 02:34:31.926047 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:31.911483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.7.194.254:44621)
0619 02:34:31.911802 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:31.911807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:31.911824 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:34:31.911953 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:31.911956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:31.912010 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:31.912101 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:31.912914 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.913736 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.914910 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:31.915274 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:31.915346 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:31.917306 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:31.917311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:31.917314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:31.917345 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:31.923915 (+ 6570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:31.924472 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:31.924476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:31.924478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:31.924576 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:31.924874 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:31.924877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:31.924890 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:31.925133 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:31.925256 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:31.925718 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 02:34:31.925866 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
W20260619 02:34:31.927033 9061 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47622
I20260619 02:34:31.929147 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 9134
I20260619 02:34:31.940668 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 9274
I20260619 02:34:31.948379 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 9414
I20260619 02:34:31.954864 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 9030
2026-06-19T02:34:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3137 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:32 dist-test-slave-9tkz krb5kdc[9654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:32 dist-test-slave-9tkz krb5kdc[9654](info): set up 2 sockets
Jun 19 02:34:32 dist-test-slave-9tkz krb5kdc[9654](info): commencing operation
krb5kdc: starting...
W20260619 02:34:32.997467 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:33 dist-test-slave-9tkz krb5kdc[9654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:33.125325 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:40879
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:40879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:33.238787 9667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:33.239121 9667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:33.239207 9667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:33.243309 9667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:33.243419 9667 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:34:33.243453 9667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:33.243495 9667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:33.243530 9667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:33.243577 9667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:33.249022 9667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:40879
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:40879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:33.250244 9667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:33.251387 9667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:33.260349 9675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:33.260335 9672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:33.260316 9673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:33.260422 9667 server_base.cc:1034] running on GCE node
I20260619 02:34:33.261241 9667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:33.262495 9667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:33.263738 9667 hybrid_clock.cc:648] HybridClock initialized: now 1781836473263706 us; error 44 us; skew 500 ppm
Jun 19 02:34:33 dist-test-slave-9tkz krb5kdc[9654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:33.266708 9667 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:33.267621 9667 webserver.cc:460] Webserver started at http://127.7.194.254:36361/ using document root <none> and password file <none>
I20260619 02:34:33.268213 9667 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:33.268317 9667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:33.268568 9667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:33.270823 9667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "878acf6dff33426db10cec9ef1ea108c"
format_stamp: "Formatted at 2026-06-19 02:34:33 on dist-test-slave-9tkz"
server_key: "1b358e20fbd31b2d24d2a308be6dc8a7"
server_key_iv: "dddbc970fdd96c3143eb00a32435ae06"
server_key_version: "encryptionkey@0"
I20260619 02:34:33.271453 9667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "878acf6dff33426db10cec9ef1ea108c"
format_stamp: "Formatted at 2026-06-19 02:34:33 on dist-test-slave-9tkz"
server_key: "1b358e20fbd31b2d24d2a308be6dc8a7"
server_key_iv: "dddbc970fdd96c3143eb00a32435ae06"
server_key_version: "encryptionkey@0"
I20260619 02:34:33.275467 9667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:33.278091 9682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:33.279409 9667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:34:33.279567 9667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "878acf6dff33426db10cec9ef1ea108c"
format_stamp: "Formatted at 2026-06-19 02:34:33 on dist-test-slave-9tkz"
server_key: "1b358e20fbd31b2d24d2a308be6dc8a7"
server_key_iv: "dddbc970fdd96c3143eb00a32435ae06"
server_key_version: "encryptionkey@0"
I20260619 02:34:33.279699 9667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:33.290901 9667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:33.294203 9667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:33.294473 9667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:33.302870 9667 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:40879
I20260619 02:34:33.302932 9744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:40879 every 8 connection(s)
I20260619 02:34:33.303929 9667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:33.306834 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:33.312093 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 9667
I20260619 02:34:33.312187 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:33.312487 7947 external_mini_cluster.cc:1428] Setting key 311fa40ad1f931070ef889229447e28d
I20260619 02:34:33.312705 9745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: Bootstrap starting.
I20260619 02:34:33.315394 9745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:33.316175 9745 log.cc:824] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:33.318202 9745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: No bootstrap required, opened a new log
Jun 19 02:34:33 dist-test-slave-9tkz krb5kdc[9654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:33.322166 9745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878acf6dff33426db10cec9ef1ea108c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40879 } }
I20260619 02:34:33.322402 9745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:33.322476 9745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878acf6dff33426db10cec9ef1ea108c, State: Initialized, Role: FOLLOWER
I20260619 02:34:33.322925 9745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878acf6dff33426db10cec9ef1ea108c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40879 } }
I20260619 02:34:33.323107 9745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:33.323179 9745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:33.323292 9745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:33.324519 9745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878acf6dff33426db10cec9ef1ea108c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40879 } }
I20260619 02:34:33.324945 9745 leader_election.cc:302] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 878acf6dff33426db10cec9ef1ea108c; no voters:
I20260619 02:34:33.325304 9745 leader_election.cc:288] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:33.325434 9750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:33.325781 9750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [term 1 LEADER]: Becoming Leader. State: Replica: 878acf6dff33426db10cec9ef1ea108c, State: Running, Role: LEADER
I20260619 02:34:33.326123 9750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878acf6dff33426db10cec9ef1ea108c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40879 } }
I20260619 02:34:33.326702 9745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:33.327469 9752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 878acf6dff33426db10cec9ef1ea108c. Latest consensus state: current_term: 1 leader_uuid: "878acf6dff33426db10cec9ef1ea108c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878acf6dff33426db10cec9ef1ea108c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40879 } } }
I20260619 02:34:33.327608 9752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:33.327970 9751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "878acf6dff33426db10cec9ef1ea108c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878acf6dff33426db10cec9ef1ea108c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40879 } } }
I20260619 02:34:33.328068 9751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:33.328999 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:33.314037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.7.194.254:40879)
0619 02:34:33.314509 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:33.314525 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:33.314561 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:34:33.315361 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:33.315406 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:33.315833 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:33.316198 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:33.317043 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.318120 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.319078 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.319561 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.319750 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:33.322647 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:33.322653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:33.322667 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:33.322720 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:33.324616 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.325177 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.325186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.325195 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.325386 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.325800 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.325805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.325808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.326526 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:33.326745 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:33.327504 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 02:34:33.327862 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":147,"threads_started":1}
W20260619 02:34:33.331877 9765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:34:33.331951 9765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:34:33.332069 9759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:33.332911 9759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:33.338646 9759 catalog_manager.cc:1347] Generated new cluster ID: d8920836cdc945d38c524de6170a7ff7
I20260619 02:34:33.338723 9759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:33.369385 9759 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:33.370819 9759 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:33.386677 9759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: Generated new TSK 0
I20260619 02:34:33.387614 9759 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:34:33.424537 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:40879
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:33.540632 9771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:33.540932 9771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:33.540985 9771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:33.545112 9771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:33.545184 9771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:33.545267 9771 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:34:33.545303 9771 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:34:33.545359 9771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:33.551097 9771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.194.254:40879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:33.552256 9771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:33.553359 9771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:33.563409 9779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:33.563409 9776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:33.563416 9777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:33.563807 9771 server_base.cc:1034] running on GCE node
I20260619 02:34:33.564193 9771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:33.564872 9771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:33.566056 9771 hybrid_clock.cc:648] HybridClock initialized: now 1781836473566041 us; error 28 us; skew 500 ppm
Jun 19 02:34:33 dist-test-slave-9tkz krb5kdc[9654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:33.568941 9771 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:34:33.569967 9771 webserver.cc:460] Webserver started at http://127.7.194.193:41231/ using document root <none> and password file <none>
I20260619 02:34:33.570573 9771 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:33.570637 9771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:33.570921 9771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:33.573014 9771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "2b9ffe8f71e24cb388303b5fb414f4d2"
format_stamp: "Formatted at 2026-06-19 02:34:33 on dist-test-slave-9tkz"
server_key: "71a5d7b8befa2f473563d864e8877067"
server_key_iv: "9daa1d1d43cda2f4cd9e1a7029f31c5c"
server_key_version: "encryptionkey@0"
I20260619 02:34:33.573654 9771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "2b9ffe8f71e24cb388303b5fb414f4d2"
format_stamp: "Formatted at 2026-06-19 02:34:33 on dist-test-slave-9tkz"
server_key: "71a5d7b8befa2f473563d864e8877067"
server_key_iv: "9daa1d1d43cda2f4cd9e1a7029f31c5c"
server_key_version: "encryptionkey@0"
I20260619 02:34:33.577409 9771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:33.579895 9786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:33.581147 9771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:34:33.581267 9771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "2b9ffe8f71e24cb388303b5fb414f4d2"
format_stamp: "Formatted at 2026-06-19 02:34:33 on dist-test-slave-9tkz"
server_key: "71a5d7b8befa2f473563d864e8877067"
server_key_iv: "9daa1d1d43cda2f4cd9e1a7029f31c5c"
server_key_version: "encryptionkey@0"
I20260619 02:34:33.581386 9771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:33.595358 9771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:33.598459 9771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:33.598673 9771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:33.600390 9771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:33.600481 9771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:33.600557 9771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:33.600610 9771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:34:33 dist-test-slave-9tkz krb5kdc[9654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:33.611848 9771 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:46703
I20260619 02:34:33.611915 9901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:46703 every 8 connection(s)
I20260619 02:34:33.613066 9771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:33.615223 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:33.601381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:45323 (local address 127.7.194.254:40879)
0619 02:34:33.601516 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:33.601519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:33.602597 (+ 1078us) server_negotiation.cc:407] Connection header received
0619 02:34:33.603779 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:33.603783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:33.603862 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:33.603971 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:33.605651 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.606422 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.607816 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.608202 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.608317 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:33.611073 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:33.611076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:33.611078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:33.611104 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:33.612864 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.613466 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.613473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.613478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.613615 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.614068 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.614071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.614073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.614276 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:33.614386 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:33.614899 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 02:34:33.615031 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:34:33.616082 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:33.601929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40879 (local address 127.7.194.193:45323)
0619 02:34:33.602435 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:33.602480 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:33.603482 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:33.604185 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:33.604194 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:33.604762 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:33.605473 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:33.605498 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.606551 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.606555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:33.607683 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:33.607697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.608367 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.608371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:33.608558 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:33.609231 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:33.609248 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:33.610883 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:33.612995 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:33.613000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:33.613013 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:33.613342 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:33.613734 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:33.613739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:33.613754 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:33.613941 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:33.614385 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:33.614392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:33.614643 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 02:34:33.614842 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 02:34:33.615174 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":127,"threads_started":1}
I20260619 02:34:33.617362 9902 heartbeater.cc:347] Connected to a master server at 127.7.194.254:40879
I20260619 02:34:33.617522 9902 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:33.618060 9902 heartbeater.cc:511] Master 127.7.194.254:40879 requested a full tablet report, sending...
I20260619 02:34:33.619792 9698 ts_manager.cc:194] Registered new tserver with Master: 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193:46703)
I20260619 02:34:33.620966 9698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:45323
I20260619 02:34:33.621830 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 9771
I20260619 02:34:33.621958 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:33.622309 7947 external_mini_cluster.cc:1428] Setting key 5b8ffd9294d0056d1f49f24ec2ad5a4d
I20260619 02:34:33.627323 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:33.619701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:60753 (local address 127.7.194.254:40879)
0619 02:34:33.619851 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:33.619858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:33.620025 (+ 167us) server_negotiation.cc:407] Connection header received
0619 02:34:33.620171 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:33.620174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:33.620220 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:33.620306 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:33.620754 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.621254 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.622248 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.622469 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.622584 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:33.624041 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:33.624046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:33.624049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:33.624079 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:33.625600 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.626035 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.626040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.626042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.626126 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.626417 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.626421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.626422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.626673 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:33.626780 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:33.627041 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 02:34:33.627159 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:34:33.627353 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:33.619652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40879 (local address 127.7.194.193:60753)
0619 02:34:33.619942 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:33.619954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:33.620070 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:33.620328 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:33.620330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:33.620546 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:33.620643 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:33.620648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.621381 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.621384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:33.622126 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:33.622134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.622585 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.622589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:33.622682 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:33.623574 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:33.623586 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:33.623901 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:33.625727 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:33.625730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:33.625733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:33.625935 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:33.626219 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:33.626223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:33.626225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:33.626319 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:33.626767 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:33.626770 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:33.626866 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 02:34:33.626977 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:34:33.627168 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":4,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:34:33.652256 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:40879
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:33.769526 9911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:33.769889 9911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:33.769940 9911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:33.774171 9911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:33.774247 9911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:33.774293 9911 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:34:33.774312 9911 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:34:33.774348 9911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:33.780117 9911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.194.254:40879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:33.781221 9911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:33.782343 9911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:33.792357 9917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:33.792464 9919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:33.792613 9916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:33.792933 9911 server_base.cc:1034] running on GCE node
I20260619 02:34:33.793314 9911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:33.793977 9911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:33.795140 9911 hybrid_clock.cc:648] HybridClock initialized: now 1781836473795105 us; error 46 us; skew 500 ppm
Jun 19 02:34:33 dist-test-slave-9tkz krb5kdc[9654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:33.797907 9911 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:34:33.798882 9911 webserver.cc:460] Webserver started at http://127.7.194.194:39031/ using document root <none> and password file <none>
I20260619 02:34:33.799418 9911 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:33.799477 9911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:33.799646 9911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:33.801798 9911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "8bba1447d9a44e2aad7f415ab8cf00ce"
format_stamp: "Formatted at 2026-06-19 02:34:33 on dist-test-slave-9tkz"
server_key: "012bf30bc528883056b436fa8169c40f"
server_key_iv: "fdd6082aee7129bbeb2496f806f61948"
server_key_version: "encryptionkey@0"
I20260619 02:34:33.802333 9911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "8bba1447d9a44e2aad7f415ab8cf00ce"
format_stamp: "Formatted at 2026-06-19 02:34:33 on dist-test-slave-9tkz"
server_key: "012bf30bc528883056b436fa8169c40f"
server_key_iv: "fdd6082aee7129bbeb2496f806f61948"
server_key_version: "encryptionkey@0"
I20260619 02:34:33.806077 9911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:34:33.808672 9926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:33.810117 9911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:33.810271 9911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "8bba1447d9a44e2aad7f415ab8cf00ce"
format_stamp: "Formatted at 2026-06-19 02:34:33 on dist-test-slave-9tkz"
server_key: "012bf30bc528883056b436fa8169c40f"
server_key_iv: "fdd6082aee7129bbeb2496f806f61948"
server_key_version: "encryptionkey@0"
I20260619 02:34:33.810398 9911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:33.834877 9911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:33.838047 9911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:33.838269 9911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:33.839895 9911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:33.839977 9911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:33.840047 9911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:33.840099 9911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:34:33 dist-test-slave-9tkz krb5kdc[9654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:33.851365 9911 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:37553
I20260619 02:34:33.851492 10041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:37553 every 8 connection(s)
I20260619 02:34:33.852340 9911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:33.854547 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:33.840682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:48369 (local address 127.7.194.254:40879)
0619 02:34:33.840839 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:33.840843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:33.841871 (+ 1028us) server_negotiation.cc:407] Connection header received
0619 02:34:33.843290 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:33.843295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:33.843361 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:33.843466 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:33.845186 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.846026 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.847402 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.847668 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.847791 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:33.850714 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:33.850718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:33.850721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:33.850750 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:33.852384 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.853075 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.853080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.853083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.853180 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.853510 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.853512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.853514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.853787 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:33.853870 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:33.854322 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 02:34:33.854425 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:34:33.855374 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:33.841155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40879 (local address 127.7.194.194:48369)
0619 02:34:33.841669 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:33.841734 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:33.842972 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:33.843695 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:33.843705 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:33.844221 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:33.845003 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:33.845027 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.846179 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.846182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:33.847254 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:33.847263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.847840 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.847844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:33.848024 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:33.848697 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:33.848754 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:33.850530 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:33.852545 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:33.852550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:33.852566 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:33.852935 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:33.853262 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:33.853265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:33.853266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:33.853412 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:33.853878 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:33.853883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:33.854111 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 02:34:33.854265 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 02:34:33.854534 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
I20260619 02:34:33.856570 10042 heartbeater.cc:347] Connected to a master server at 127.7.194.254:40879
I20260619 02:34:33.856727 10042 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:33.857256 10042 heartbeater.cc:511] Master 127.7.194.254:40879 requested a full tablet report, sending...
I20260619 02:34:33.858354 9699 ts_manager.cc:194] Registered new tserver with Master: 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553)
I20260619 02:34:33.859131 9699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:48369
I20260619 02:34:33.859143 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 9911
I20260619 02:34:33.859282 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:33.859625 7947 external_mini_cluster.cc:1428] Setting key 2b01d921ef02a21a7c9e1cd0ab43ee25
I20260619 02:34:33.867851 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:33.859052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:36651 (local address 127.7.194.254:40879)
0619 02:34:33.859173 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:33.859177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:33.859662 (+ 485us) server_negotiation.cc:407] Connection header received
0619 02:34:33.859805 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:33.859808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:33.859857 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:33.859942 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:33.860871 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.861407 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.862724 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.863108 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.863232 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:33.864534 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:33.864538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:33.864540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:33.864560 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:33.866086 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.866599 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.866602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.866603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.866677 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:33.866998 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:33.867001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:33.867002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:33.867204 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:33.867285 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:33.867600 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 02:34:33.867697 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:34:33.867954 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:33.858966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40879 (local address 127.7.194.194:36651)
0619 02:34:33.859567 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:33.859582 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:33.859680 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:33.859973 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:33.859976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:33.860202 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:33.860739 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:33.860748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.861532 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.861538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:33.862583 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:33.862597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:33.863229 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:33.863232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:33.863304 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:33.863870 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:33.863890 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:33.864404 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:33.866202 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:33.866208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:33.866212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:33.866493 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:33.866774 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:33.866780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:33.866785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:33.866902 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:33.867307 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:33.867312 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:33.867432 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 02:34:33.867556 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 02:34:33.867716 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:34:33.888455 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:40879
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:34.002857 10051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:34.003185 10051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:34.003275 10051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:34.007398 10051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:34.007498 10051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:34.007568 10051 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:34:34.007615 10051 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:34:34.007690 10051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:34.013263 10051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.194.254:40879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:34.014557 10051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:34.015707 10051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:34.025888 10059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:34.025902 10057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:34.025998 10056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:34.026160 10051 server_base.cc:1034] running on GCE node
I20260619 02:34:34.026692 10051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:34.027522 10051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:34.028688 10051 hybrid_clock.cc:648] HybridClock initialized: now 1781836474028658 us; error 47 us; skew 500 ppm
Jun 19 02:34:34 dist-test-slave-9tkz krb5kdc[9654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836474, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:34.031708 10051 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:34:34.032618 10051 webserver.cc:460] Webserver started at http://127.7.194.195:34407/ using document root <none> and password file <none>
I20260619 02:34:34.033223 10051 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:34.033291 10051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:34.033631 10051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:34.035970 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "1442a84ea91b4a5e9763f165397f4519"
format_stamp: "Formatted at 2026-06-19 02:34:34 on dist-test-slave-9tkz"
server_key: "ea09b0fc75de7a749c5970ffba1e9bbe"
server_key_iv: "cf59d98be9e310ddc83dc40d6db3ac40"
server_key_version: "encryptionkey@0"
I20260619 02:34:34.036557 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "1442a84ea91b4a5e9763f165397f4519"
format_stamp: "Formatted at 2026-06-19 02:34:34 on dist-test-slave-9tkz"
server_key: "ea09b0fc75de7a749c5970ffba1e9bbe"
server_key_iv: "cf59d98be9e310ddc83dc40d6db3ac40"
server_key_version: "encryptionkey@0"
I20260619 02:34:34.040413 10051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 02:34:34.042995 10066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:34.044270 10051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:34.044395 10051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "1442a84ea91b4a5e9763f165397f4519"
format_stamp: "Formatted at 2026-06-19 02:34:34 on dist-test-slave-9tkz"
server_key: "ea09b0fc75de7a749c5970ffba1e9bbe"
server_key_iv: "cf59d98be9e310ddc83dc40d6db3ac40"
server_key_version: "encryptionkey@0"
I20260619 02:34:34.044516 10051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:34.063328 10051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:34.066602 10051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:34.066823 10051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:34.068336 10051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:34.068385 10051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:34.068425 10051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:34.068486 10051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:34:34 dist-test-slave-9tkz krb5kdc[9654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836474, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:34.079388 10051 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:41339
I20260619 02:34:34.079694 10181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:41339 every 8 connection(s)
I20260619 02:34:34.080677 10051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:34.083272 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.069439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:37919 (local address 127.7.194.254:40879)
0619 02:34:34.069628 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.069633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.070560 (+ 927us) server_negotiation.cc:407] Connection header received
0619 02:34:34.071876 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.071880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.071941 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.072036 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:34.073807 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.074734 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.076096 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.076419 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.076549 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.079274 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:34.079280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:34.079287 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:34.079323 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:34.081082 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:34.081780 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:34.081786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:34.081791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:34.081900 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:34.082248 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:34.082251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:34.082252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:34.082436 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:34.082510 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.082993 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.083148 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:34:34.084089 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.069909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40879 (local address 127.7.194.195:37919)
0619 02:34:34.070379 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.070431 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.071615 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.072257 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.072268 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.072806 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:34.073630 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.073656 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.074862 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.074886 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.075958 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.075968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.076615 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.076620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.076802 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.077475 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:34.077492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:34.079086 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:34.081215 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:34.081220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:34.081235 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:34.081656 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:34.081994 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:34.081997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:34.081998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:34.082143 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:34.082524 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:34.082529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:34.082777 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.082925 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.083181 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":131,"threads_started":1}
I20260619 02:34:34.085271 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 10051
I20260619 02:34:34.085399 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:34.085610 10182 heartbeater.cc:347] Connected to a master server at 127.7.194.254:40879
I20260619 02:34:34.085753 10182 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:34.085786 7947 external_mini_cluster.cc:1428] Setting key c0239ad65ff4505eb6735ad59034b194
I20260619 02:34:34.086350 10182 heartbeater.cc:511] Master 127.7.194.254:40879 requested a full tablet report, sending...
I20260619 02:34:34.087450 9699 ts_manager.cc:194] Registered new tserver with Master: 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:34.088219 9699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:37919
I20260619 02:34:34.094782 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.087898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40879 (local address 127.7.194.195:52445)
0619 02:34:34.088145 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.088159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.088237 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.088470 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.088476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.088705 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:34.088832 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.088839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.089839 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.089842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.090401 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.090408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.090862 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.090864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.090929 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.091396 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:34.091406 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:34.091712 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:34.093504 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:34.093506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:34.093508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:34.093748 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:34.093988 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:34.093990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:34.093992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:34.094060 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:34.094386 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:34.094388 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:34.094459 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.094535 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.094639 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260619 02:34:34.094815 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.087937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:52445 (local address 127.7.194.254:40879)
0619 02:34:34.088044 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.088048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.088234 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:34:34.088316 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.088319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.088360 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.088437 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:34.088955 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.089715 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.090516 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.090722 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.090845 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.091858 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:34.091861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:34.091863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:34.091884 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:34.093412 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:34.093832 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:34.093835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:34.093836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:34.093909 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:34.094136 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:34.094139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:34.094140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:34.094298 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:34.094374 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.094567 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.094674 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:34:34.099737 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:34:34 dist-test-slave-9tkz krb5kdc[9654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:34:34 dist-test-slave-9tkz krb5kdc[9654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:34.125690 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.117595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.7.194.254:40879)
0619 02:34:34.117731 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.117734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.117748 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:34.117792 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.117801 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.117844 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.117923 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:34.118697 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.119239 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.120063 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.120268 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.120342 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.122735 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:34.122738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:34.122740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:34.122766 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:34.124245 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:34.124606 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:34.124609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:34.124610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:34.124685 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:34.124930 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:34.124932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:34.124934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:34.125108 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:34.125206 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.125403 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.125497 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:34:34.128818 9699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:34:34.131914 9699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:34.146433 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.140565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.7.194.194:37553)
0619 02:34:34.141395 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.141405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.141429 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:34:34.141510 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.141517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.141759 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.142302 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:34.142430 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.143572 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.144999 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.145749 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.145993 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.146133 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.146186 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.146282 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":117,"threads_started":1}
I20260619 02:34:34.146422 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.140634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46703 (local address 127.0.0.1:33868)
0619 02:34:34.141324 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.141336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.141413 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.142133 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.142136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.142154 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:34.142238 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.142242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.143368 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.143371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.144747 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.144757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.145765 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.145780 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.145917 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.145985 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.146095 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.146258 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":40,"thread_start_us":91,"threads_started":1}
I20260619 02:34:34.146440 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.140419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:37553 (local address 127.0.0.1:47208)
0619 02:34:34.141182 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.141237 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.141382 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.141962 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.141968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.141995 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:34.142144 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.142151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.143690 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.143693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.144881 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.144890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.145878 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.145884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.146020 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.146044 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.146114 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.146268 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":117,"thread_start_us":79,"threads_started":1}
I20260619 02:34:34.146859 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.141182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:41339 (local address 127.0.0.1:43388)
0619 02:34:34.141809 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.141822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.141909 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.142701 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.142704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.142722 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:34.142794 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.142800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.143991 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.143996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.145190 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.145200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.146452 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.146455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.146550 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.146575 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.146646 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.146717 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":55,"thread_start_us":111,"threads_started":1}
I20260619 02:34:34.147109 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.140760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.7.194.193:46703)
0619 02:34:34.141203 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.141211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.141516 (+ 305us) server_negotiation.cc:407] Connection header received
0619 02:34:34.141619 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.141624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.141823 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.141980 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:34.142352 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.143263 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.144900 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.145645 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.145752 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.145908 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.146832 (+ 924us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.146971 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":65,"threads_started":1}
I20260619 02:34:34.148013 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.141711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.7.194.195:41339)
0619 02:34:34.142285 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.142296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.142318 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:34:34.142386 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.142390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.142558 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.142704 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:34.142911 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.143884 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.145318 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.146155 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.146251 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.146393 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.147057 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.147148 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":95,"threads_started":1}
I20260619 02:34:34.149883 9836 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6ce49c38944c5b808c2ad87dcf095d (DEFAULT_TABLE table=test-table [id=48af310be8f94e1385d6402bf934364c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:34.149880 9976 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6ce49c38944c5b808c2ad87dcf095d (DEFAULT_TABLE table=test-table [id=48af310be8f94e1385d6402bf934364c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:34.150810 9836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6ce49c38944c5b808c2ad87dcf095d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:34.150470 10116 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6ce49c38944c5b808c2ad87dcf095d (DEFAULT_TABLE table=test-table [id=48af310be8f94e1385d6402bf934364c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:34.150893 9976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6ce49c38944c5b808c2ad87dcf095d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:34.151309 10116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6ce49c38944c5b808c2ad87dcf095d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:34.156760 10205 tablet_bootstrap.cc:492] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: Bootstrap starting.
I20260619 02:34:34.159072 10207 tablet_bootstrap.cc:492] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: Bootstrap starting.
I20260619 02:34:34.159377 10206 tablet_bootstrap.cc:492] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: Bootstrap starting.
I20260619 02:34:34.159855 10205 tablet_bootstrap.cc:654] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:34.160795 10205 log.cc:824] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:34.161304 10207 tablet_bootstrap.cc:654] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:34.161497 10206 tablet_bootstrap.cc:654] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:34.162248 10206 log.cc:824] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:34.162186 10207 log.cc:824] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:34.162971 10205 tablet_bootstrap.cc:492] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: No bootstrap required, opened a new log
I20260619 02:34:34.163172 10205 ts_tablet_manager.cc:1397] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 02:34:34.165757 10207 tablet_bootstrap.cc:492] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: No bootstrap required, opened a new log
I20260619 02:34:34.165757 10206 tablet_bootstrap.cc:492] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: No bootstrap required, opened a new log
I20260619 02:34:34.165961 10206 ts_tablet_manager.cc:1397] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260619 02:34:34.165962 10207 ts_tablet_manager.cc:1397] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 02:34:34.167006 10205 raft_consensus.cc:348] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.167259 10205 raft_consensus.cc:374] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:34.167322 10205 raft_consensus.cc:729] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9ffe8f71e24cb388303b5fb414f4d2, State: Initialized, Role: FOLLOWER
I20260619 02:34:34.167785 10205 consensus_queue.cc:260] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.168433 9902 heartbeater.cc:503] Master 127.7.194.254:40879 was elected leader, sending a full tablet report...
I20260619 02:34:34.168812 10205 ts_tablet_manager.cc:1428] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:34:34.169937 10207 raft_consensus.cc:348] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.170209 10207 raft_consensus.cc:374] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:34.170083 10206 raft_consensus.cc:348] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.170287 10206 raft_consensus.cc:374] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:34.170264 10207 raft_consensus.cc:729] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1442a84ea91b4a5e9763f165397f4519, State: Initialized, Role: FOLLOWER
I20260619 02:34:34.170342 10206 raft_consensus.cc:729] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bba1447d9a44e2aad7f415ab8cf00ce, State: Initialized, Role: FOLLOWER
I20260619 02:34:34.170768 10207 consensus_queue.cc:260] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.170775 10206 consensus_queue.cc:260] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.171657 10042 heartbeater.cc:503] Master 127.7.194.254:40879 was elected leader, sending a full tablet report...
I20260619 02:34:34.171663 10182 heartbeater.cc:503] Master 127.7.194.254:40879 was elected leader, sending a full tablet report...
I20260619 02:34:34.171762 10207 ts_tablet_manager.cc:1428] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:34:34.171762 10206 ts_tablet_manager.cc:1428] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:34:34.179932 10213 raft_consensus.cc:482] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:34.180119 10213 raft_consensus.cc:504] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.181226 10213 leader_election.cc:288] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193:46703), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:34.186157 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.181662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:41339 (local address 127.7.194.194:51019)
0619 02:34:34.182064 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.182079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.182186 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.182418 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.182421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.182436 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:34.182567 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.182572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.183356 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.183359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.184576 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.184588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.185640 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.185643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.185755 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.185812 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.185960 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.186025 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":15,"thread_start_us":148,"threads_started":1}
I20260619 02:34:34.186271 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.181983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:51019 (local address 127.7.194.195:41339)
0619 02:34:34.182137 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.182142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.182156 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:34.182271 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.182274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.182321 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.182408 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:34.182703 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.183251 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.184797 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.185426 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.185601 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.185664 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.186072 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.186151 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:34:34.186317 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.181411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46703 (local address 127.7.194.194:33477)
0619 02:34:34.181632 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.181649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.181736 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.182100 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.182103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.182123 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:34.182279 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.182285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.183106 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.183113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.184733 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.184746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.185769 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.185772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.185918 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.185961 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.186035 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.186115 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"spinlock_wait_cycles":66816}
I20260619 02:34:34.186419 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.181688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:33477 (local address 127.7.194.193:46703)
0619 02:34:34.181837 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.181842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.181860 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:34:34.181921 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.181925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.181973 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.182063 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:34.182407 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.182991 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.184907 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.185532 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.185658 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.185710 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.186207 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.186306 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:34:34.187019 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6ce49c38944c5b808c2ad87dcf095d" candidate_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519" is_pre_election: true
I20260619 02:34:34.187017 9856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6ce49c38944c5b808c2ad87dcf095d" candidate_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" is_pre_election: true
I20260619 02:34:34.187327 9856 raft_consensus.cc:2454] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bba1447d9a44e2aad7f415ab8cf00ce in term 0.
I20260619 02:34:34.187341 10136 raft_consensus.cc:2454] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bba1447d9a44e2aad7f415ab8cf00ce in term 0.
I20260619 02:34:34.187781 9930 leader_election.cc:302] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1442a84ea91b4a5e9763f165397f4519, 2b9ffe8f71e24cb388303b5fb414f4d2, 8bba1447d9a44e2aad7f415ab8cf00ce; no voters:
I20260619 02:34:34.188076 10213 raft_consensus.cc:2781] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:34.188166 10213 raft_consensus.cc:482] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:34.188235 10213 raft_consensus.cc:3037] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:34.189472 10213 raft_consensus.cc:504] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.189927 10213 leader_election.cc:288] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [CANDIDATE]: Term 1 election: Requested vote from peers 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193:46703), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:34.190275 9856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6ce49c38944c5b808c2ad87dcf095d" candidate_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2"
I20260619 02:34:34.190390 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6ce49c38944c5b808c2ad87dcf095d" candidate_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519"
I20260619 02:34:34.190425 9856 raft_consensus.cc:3037] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:34.190582 10136 raft_consensus.cc:3037] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:34.191569 9856 raft_consensus.cc:2454] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bba1447d9a44e2aad7f415ab8cf00ce in term 1.
I20260619 02:34:34.191658 10136 raft_consensus.cc:2454] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bba1447d9a44e2aad7f415ab8cf00ce in term 1.
I20260619 02:34:34.191918 9930 leader_election.cc:302] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9ffe8f71e24cb388303b5fb414f4d2, 8bba1447d9a44e2aad7f415ab8cf00ce; no voters:
I20260619 02:34:34.192135 10213 raft_consensus.cc:2781] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:34.192404 10213 raft_consensus.cc:686] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 LEADER]: Becoming Leader. State: Replica: 8bba1447d9a44e2aad7f415ab8cf00ce, State: Running, Role: LEADER
I20260619 02:34:34.192780 10213 consensus_queue.cc:237] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.196117 9698 catalog_manager.cc:5495] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194). New cstate: current_term: 1 leader_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:34:34 dist-test-slave-9tkz krb5kdc[9654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:34.243602 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.235645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40879 (local address 127.0.0.1:35458)
0619 02:34:34.235920 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.235936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.236054 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.236300 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.236302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.236699 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:34.236836 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.236841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.237543 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.237585 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.238124 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.238132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.238538 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.238540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.238599 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.239176 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:34.239188 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:34.240627 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:34.242159 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:34.242166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:34.242171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:34.242452 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:34.242686 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:34.242688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:34.242689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:34.242771 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:34.243110 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:34.243116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:34.243218 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.243313 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.243434 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":114,"threads_started":1}
I20260619 02:34:34.243610 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.235673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.7.194.254:40879)
0619 02:34:34.235791 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.235795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.236017 (+ 222us) server_negotiation.cc:407] Connection header received
0619 02:34:34.236120 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.236122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.236161 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.236258 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:34.236939 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.237440 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.238243 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.238441 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.238516 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.240755 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:34.240759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:34.240761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:34.240780 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:34.242058 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:34.242538 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:34.242541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:34.242542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:34.242608 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:34.242845 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:34.242847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:34.242848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:34.243019 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:34.243090 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.243367 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.243487 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":44}
I20260619 02:34:34.247570 9698 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:35458:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:34:34.248466 9698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:34.253887 9836 tablet_service.cc:1467] Processing CreateTablet for tablet b9d5d42c57844900bb7792a37b7ddf49 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=02e149d8734e4cd0b79dbc6053b4c612]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:34:34.253888 9976 tablet_service.cc:1467] Processing CreateTablet for tablet b9d5d42c57844900bb7792a37b7ddf49 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=02e149d8734e4cd0b79dbc6053b4c612]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:34:34.254693 9836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d5d42c57844900bb7792a37b7ddf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:34.254691 9976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d5d42c57844900bb7792a37b7ddf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:34.257812 10205 tablet_bootstrap.cc:492] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: Bootstrap starting.
I20260619 02:34:34.257812 10206 tablet_bootstrap.cc:492] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: Bootstrap starting.
I20260619 02:34:34.258183 10116 tablet_service.cc:1467] Processing CreateTablet for tablet b9d5d42c57844900bb7792a37b7ddf49 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=02e149d8734e4cd0b79dbc6053b4c612]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:34:34.258855 10205 tablet_bootstrap.cc:654] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:34.258855 10206 tablet_bootstrap.cc:654] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:34.258908 10116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d5d42c57844900bb7792a37b7ddf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:34.260330 10205 tablet_bootstrap.cc:492] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: No bootstrap required, opened a new log
I20260619 02:34:34.260434 10205 ts_tablet_manager.cc:1397] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:34:34.260574 10206 tablet_bootstrap.cc:492] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: No bootstrap required, opened a new log
I20260619 02:34:34.260672 10206 ts_tablet_manager.cc:1397] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 02:34:34.260912 10205 raft_consensus.cc:348] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.261039 10205 raft_consensus.cc:374] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:34.261080 10205 raft_consensus.cc:729] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9ffe8f71e24cb388303b5fb414f4d2, State: Initialized, Role: FOLLOWER
I20260619 02:34:34.261078 10206 raft_consensus.cc:348] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.261197 10206 raft_consensus.cc:374] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:34.261237 10206 raft_consensus.cc:729] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bba1447d9a44e2aad7f415ab8cf00ce, State: Initialized, Role: FOLLOWER
I20260619 02:34:34.261214 10205 consensus_queue.cc:260] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.261375 10206 consensus_queue.cc:260] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.261495 10205 ts_tablet_manager.cc:1428] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:34.261689 10213 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.261809 10213 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:34.261705 10211 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.261860 10211 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:34.262066 10207 tablet_bootstrap.cc:492] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: Bootstrap starting.
I20260619 02:34:34.262359 10206 ts_tablet_manager.cc:1428] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:34:34.263141 10207 tablet_bootstrap.cc:654] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:34.264467 10207 tablet_bootstrap.cc:492] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: No bootstrap required, opened a new log
I20260619 02:34:34.264554 10207 ts_tablet_manager.cc:1397] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:34:34.264968 10207 raft_consensus.cc:348] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.265100 10207 raft_consensus.cc:374] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:34.265134 10207 raft_consensus.cc:729] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1442a84ea91b4a5e9763f165397f4519, State: Initialized, Role: FOLLOWER
I20260619 02:34:34.265264 10207 consensus_queue.cc:260] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.265528 10207 ts_tablet_manager.cc:1428] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:34:34.265513 10212 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.265727 10212 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 02:34:34.332415 10183 tablet.cc:2369] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:34:34.353711 10043 tablet.cc:2369] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:34:34.364581 9903 tablet.cc:2369] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:34:34.556702 10213 raft_consensus.cc:482] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:34.556900 10213 raft_consensus.cc:504] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.557443 10213 leader_election.cc:288] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193:46703), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:34.558025 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5d42c57844900bb7792a37b7ddf49" candidate_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519" is_pre_election: true
I20260619 02:34:34.558022 9856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5d42c57844900bb7792a37b7ddf49" candidate_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" is_pre_election: true
I20260619 02:34:34.558215 10136 raft_consensus.cc:2454] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bba1447d9a44e2aad7f415ab8cf00ce in term 0.
I20260619 02:34:34.558215 9856 raft_consensus.cc:2454] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bba1447d9a44e2aad7f415ab8cf00ce in term 0.
I20260619 02:34:34.558561 9930 leader_election.cc:302] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9ffe8f71e24cb388303b5fb414f4d2, 8bba1447d9a44e2aad7f415ab8cf00ce; no voters:
I20260619 02:34:34.558821 10213 raft_consensus.cc:2781] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:34.558877 10213 raft_consensus.cc:482] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:34.558913 10213 raft_consensus.cc:3037] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:34.559959 10213 raft_consensus.cc:504] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.560395 10213 leader_election.cc:288] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [CANDIDATE]: Term 1 election: Requested vote from peers 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193:46703), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:34.560806 9856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5d42c57844900bb7792a37b7ddf49" candidate_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2"
I20260619 02:34:34.560949 9856 raft_consensus.cc:3037] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:34.560994 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5d42c57844900bb7792a37b7ddf49" candidate_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519"
I20260619 02:34:34.561157 10136 raft_consensus.cc:3037] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:34.562121 9856 raft_consensus.cc:2454] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bba1447d9a44e2aad7f415ab8cf00ce in term 1.
I20260619 02:34:34.562198 10136 raft_consensus.cc:2454] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bba1447d9a44e2aad7f415ab8cf00ce in term 1.
I20260619 02:34:34.562470 9930 leader_election.cc:302] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9ffe8f71e24cb388303b5fb414f4d2, 8bba1447d9a44e2aad7f415ab8cf00ce; no voters:
I20260619 02:34:34.562638 10213 raft_consensus.cc:2781] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:34.562753 10213 raft_consensus.cc:686] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 LEADER]: Becoming Leader. State: Replica: 8bba1447d9a44e2aad7f415ab8cf00ce, State: Running, Role: LEADER
I20260619 02:34:34.562896 10213 consensus_queue.cc:237] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.563617 10215 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: TxnStatusTablet state changed. Reason: New leader 8bba1447d9a44e2aad7f415ab8cf00ce. Latest consensus state: current_term: 1 leader_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.563735 10215 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:34:34.564010 10227 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:34:34.564276 10227 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:34:34.564778 9697 catalog_manager.cc:5495] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194). New cstate: current_term: 1 leader_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } health_report { overall_health: UNKNOWN } } }
I20260619 02:34:34.604177 10215 consensus_queue.cc:1035] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:34.610706 10215 consensus_queue.cc:1035] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:34.629218 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.625216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.7.194.194:37553)
0619 02:34:34.625333 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.625337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.625353 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:34:34.625512 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.625515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.625605 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.625697 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:34.626097 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.626602 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.627676 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.627908 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.628023 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.628351 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:34.628819 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:34.628951 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.629025 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.629080 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:34:34.629217 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.625094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:37553 (local address 127.0.0.1:47218)
0619 02:34:34.625244 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.625261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.625367 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.625826 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.625829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.625835 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:34:34.625962 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.625969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.626725 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.626732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.627557 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.627566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.628031 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.628034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.628135 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.628176 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:34:34.628934 (+ 758us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:34:34.628949 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.629016 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.629097 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 02:34:34.634182 9856 raft_consensus.cc:1261] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Refusing update from remote peer 8bba1447d9a44e2aad7f415ab8cf00ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:34.634191 10136 raft_consensus.cc:1261] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Refusing update from remote peer 8bba1447d9a44e2aad7f415ab8cf00ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:34.634639 10213 consensus_queue.cc:1035] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:34.634799 10215 consensus_queue.cc:1035] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:34.637498 10234 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: TxnStatusTablet state changed. Reason: New leader 8bba1447d9a44e2aad7f415ab8cf00ce. Latest consensus state: current_term: 1 leader_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.637666 10234 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:34.637701 10211 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: TxnStatusTablet state changed. Reason: New leader 8bba1447d9a44e2aad7f415ab8cf00ce. Latest consensus state: current_term: 1 leader_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.637804 10211 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:34.640961 10230 mvcc.cc:205] Tried to move back new op lower bound from 7298402200096223232 to 7298402199811190784. Current Snapshot: MvccSnapshot[applied={T|T < 7298402200096223232}]
I20260619 02:34:34.641085 10213 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.641197 10213 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:34:34.641394 10211 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.641508 10211 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 2b9ffe8f71e24cb388303b5fb414f4d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:34.643421 10215 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.643525 10215 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 8bba1447d9a44e2aad7f415ab8cf00ce: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:34:34.644847 10233 mvcc.cc:205] Tried to move back new op lower bound from 7298402200096223232 to 7298402199811190784. Current Snapshot: MvccSnapshot[applied={T|T < 7298402200096223232}]
I20260619 02:34:34.645200 10234 tablet_replica.cc:439] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } } }
I20260619 02:34:34.645314 10234 tablet_replica.cc:442] T b9d5d42c57844900bb7792a37b7ddf49 P 1442a84ea91b4a5e9763f165397f4519: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:34:34.653671 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.649956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.7.194.194:37553)
0619 02:34:34.650078 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.650082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.650284 (+ 202us) server_negotiation.cc:407] Connection header received
0619 02:34:34.650414 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.650416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.650471 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.650553 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:34.650824 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.651319 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.652559 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.652791 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.652912 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.653170 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:34.653238 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:34.653291 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.653413 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.653459 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:34:34.665395 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.661767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:53359 (local address 127.7.194.194:37553)
0619 02:34:34.661868 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.661873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.662122 (+ 249us) server_negotiation.cc:407] Connection header received
0619 02:34:34.662223 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.662226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.662272 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.662347 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:34.662540 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.663254 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.664178 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.664519 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.664617 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.664950 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:34.665031 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:34.665104 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.665194 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.665241 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":25}
I20260619 02:34:34.665398 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.661749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:37553 (local address 127.7.194.194:53359)
0619 02:34:34.662039 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.662052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.662140 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.662365 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.662367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.662373 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:34:34.662476 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.662480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.663362 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.663364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.664062 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.664070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.664622 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.664624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.664718 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.664789 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:34:34.665113 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:34:34.665118 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.665166 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.665232 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":2,"thread_start_us":132,"threads_started":1}
I20260619 02:34:34.710915 9697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:34.711100 9697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:34.713740 9697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: Sending DeleteTablet for 3 replicas of tablet 6e6ce49c38944c5b808c2ad87dcf095d
I20260619 02:34:34.714582 9836 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6ce49c38944c5b808c2ad87dcf095d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:34 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:33868
I20260619 02:34:34.714551 9976 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6ce49c38944c5b808c2ad87dcf095d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:34 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:47208
I20260619 02:34:34.714655 10116 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6ce49c38944c5b808c2ad87dcf095d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:34 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:43388
I20260619 02:34:34.715062 10248 tablet_replica.cc:330] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: stopping tablet replica
I20260619 02:34:34.715173 10249 tablet_replica.cc:330] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: stopping tablet replica
I20260619 02:34:34.715404 10248 raft_consensus.cc:2229] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:34.715451 10249 raft_consensus.cc:2229] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:34:34.715669 10248 raft_consensus.cc:2258] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:34.715828 10249 raft_consensus.cc:2258] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:34.716183 9697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:34:34.716651 9697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:34.717075 10249 ts_tablet_manager.cc:1905] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:34.717075 10248 ts_tablet_manager.cc:1905] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:34.717973 10250 tablet_replica.cc:330] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: stopping tablet replica
I20260619 02:34:34.718652 10250 raft_consensus.cc:2229] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:34.718935 10250 raft_consensus.cc:2258] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:34.720444 10249 ts_tablet_manager.cc:1918] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:34:34.720551 10249 log.cc:1196] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/6e6ce49c38944c5b808c2ad87dcf095d
I20260619 02:34:34.720885 10249 ts_tablet_manager.cc:1939] T 6e6ce49c38944c5b808c2ad87dcf095d P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting consensus metadata
I20260619 02:34:34.722137 9685 catalog_manager.cc:4841] TS 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553): tablet 6e6ce49c38944c5b808c2ad87dcf095d (table test-table [id=48af310be8f94e1385d6402bf934364c]) successfully deleted
I20260619 02:34:34.722432 9836 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae6f4b075784dfa99edeffedb07d42c (DEFAULT_TABLE table=test-table [id=804ad716b7cc4a54887f38d7722e15f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:34.722713 9836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae6f4b075784dfa99edeffedb07d42c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:34.723160 10116 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae6f4b075784dfa99edeffedb07d42c (DEFAULT_TABLE table=test-table [id=804ad716b7cc4a54887f38d7722e15f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:34.723428 10116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae6f4b075784dfa99edeffedb07d42c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:34.725260 10205 tablet_bootstrap.cc:492] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: Bootstrap starting.
I20260619 02:34:34.725500 9976 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae6f4b075784dfa99edeffedb07d42c (DEFAULT_TABLE table=test-table [id=804ad716b7cc4a54887f38d7722e15f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:34.725641 10248 ts_tablet_manager.cc:1918] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:34:34.725732 10248 log.cc:1196] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/6e6ce49c38944c5b808c2ad87dcf095d
I20260619 02:34:34.725773 9976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae6f4b075784dfa99edeffedb07d42c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:34.725939 10207 tablet_bootstrap.cc:492] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: Bootstrap starting.
I20260619 02:34:34.726064 10248 ts_tablet_manager.cc:1939] T 6e6ce49c38944c5b808c2ad87dcf095d P 2b9ffe8f71e24cb388303b5fb414f4d2: Deleting consensus metadata
I20260619 02:34:34.726291 10205 tablet_bootstrap.cc:654] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:34.726897 10207 tablet_bootstrap.cc:654] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:34.727581 9686 catalog_manager.cc:4841] TS 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193:46703): tablet 6e6ce49c38944c5b808c2ad87dcf095d (table test-table [id=48af310be8f94e1385d6402bf934364c]) successfully deleted
I20260619 02:34:34.728055 10250 ts_tablet_manager.cc:1905] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:34.728281 10206 tablet_bootstrap.cc:492] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: Bootstrap starting.
I20260619 02:34:34.729182 10206 tablet_bootstrap.cc:654] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:34.730422 10206 tablet_bootstrap.cc:492] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: No bootstrap required, opened a new log
I20260619 02:34:34.730533 10206 ts_tablet_manager.cc:1397] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:34.730993 10206 raft_consensus.cc:348] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.731137 10206 raft_consensus.cc:374] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:34.731180 10206 raft_consensus.cc:729] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bba1447d9a44e2aad7f415ab8cf00ce, State: Initialized, Role: FOLLOWER
I20260619 02:34:34.731231 10250 ts_tablet_manager.cc:1918] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:34:34.731326 10250 log.cc:1196] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/6e6ce49c38944c5b808c2ad87dcf095d
I20260619 02:34:34.731310 10206 consensus_queue.cc:260] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.731633 10206 ts_tablet_manager.cc:1428] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 02:34:34.731663 10250 ts_tablet_manager.cc:1939] T 6e6ce49c38944c5b808c2ad87dcf095d P 1442a84ea91b4a5e9763f165397f4519: Deleting consensus metadata
I20260619 02:34:34.732971 9684 catalog_manager.cc:4841] TS 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339): tablet 6e6ce49c38944c5b808c2ad87dcf095d (table test-table [id=48af310be8f94e1385d6402bf934364c]) successfully deleted
I20260619 02:34:34.733731 10207 tablet_bootstrap.cc:492] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: No bootstrap required, opened a new log
I20260619 02:34:34.733829 10207 ts_tablet_manager.cc:1397] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260619 02:34:34.733943 10205 tablet_bootstrap.cc:492] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: No bootstrap required, opened a new log
I20260619 02:34:34.734036 10205 ts_tablet_manager.cc:1397] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20260619 02:34:34.734321 10207 raft_consensus.cc:348] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.734436 10207 raft_consensus.cc:374] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:34.734465 10207 raft_consensus.cc:729] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1442a84ea91b4a5e9763f165397f4519, State: Initialized, Role: FOLLOWER
I20260619 02:34:34.734513 10205 raft_consensus.cc:348] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.734583 10207 consensus_queue.cc:260] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.734639 10205 raft_consensus.cc:374] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:34.734681 10205 raft_consensus.cc:729] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9ffe8f71e24cb388303b5fb414f4d2, State: Initialized, Role: FOLLOWER
I20260619 02:34:34.734838 10207 ts_tablet_manager.cc:1428] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:34:34.734797 10205 consensus_queue.cc:260] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.735076 10205 ts_tablet_manager.cc:1428] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:34:34.949141 10211 raft_consensus.cc:482] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:34.949319 10211 raft_consensus.cc:504] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.950629 10211 leader_election.cc:288] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:34.954556 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.950976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:50007 (local address 127.7.194.194:37553)
0619 02:34:34.951150 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.951156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.951252 (+ 96us) server_negotiation.cc:407] Connection header received
0619 02:34:34.951500 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.951503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.951561 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.951651 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:34.951929 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.952443 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.953539 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.953984 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.954125 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.954189 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.954354 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.954431 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 02:34:34.954571 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.950870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:37553 (local address 127.7.194.193:50007)
0619 02:34:34.951149 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.951166 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.951345 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.951662 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.951666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.951682 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:34.951811 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.951817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.952560 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.952566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.953408 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.953416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.954112 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.954115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.954214 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.954244 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.954336 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.954419 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260619 02:34:34.954890 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.951197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:47141 (local address 127.7.194.195:41339)
0619 02:34:34.951491 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.951508 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.951528 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:34:34.951593 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.951596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.951654 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.951795 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:34.952091 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.952650 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.953757 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.954192 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.954359 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.954417 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.954614 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.954727 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260619 02:34:34.955209 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.951021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:41339 (local address 127.7.194.193:47141)
0619 02:34:34.951380 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.951396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.951501 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.951806 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.951809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.951828 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:34.951964 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.951970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.952803 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.952807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.953613 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.953622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.954357 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.954362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.954466 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.954491 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.954597 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.955007 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":21,"thread_start_us":84,"threads_started":1}
I20260619 02:34:34.955188 9995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae6f4b075784dfa99edeffedb07d42c" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" is_pre_election: true
I20260619 02:34:34.955459 9995 raft_consensus.cc:2454] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 0.
I20260619 02:34:34.955739 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae6f4b075784dfa99edeffedb07d42c" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519" is_pre_election: true
I20260619 02:34:34.955900 10136 raft_consensus.cc:2454] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 0.
I20260619 02:34:34.955905 9789 leader_election.cc:302] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9ffe8f71e24cb388303b5fb414f4d2, 8bba1447d9a44e2aad7f415ab8cf00ce; no voters:
I20260619 02:34:34.956213 10211 raft_consensus.cc:2781] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:34.956290 10211 raft_consensus.cc:482] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:34.956331 10211 raft_consensus.cc:3037] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:34.957459 10211 raft_consensus.cc:504] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.957878 10211 leader_election.cc:288] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 election: Requested vote from peers 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:34.958216 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae6f4b075784dfa99edeffedb07d42c" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519"
I20260619 02:34:34.958223 9995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae6f4b075784dfa99edeffedb07d42c" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce"
I20260619 02:34:34.958364 10136 raft_consensus.cc:3037] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:34.958374 9995 raft_consensus.cc:3037] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:34.959345 9995 raft_consensus.cc:2454] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 1.
I20260619 02:34:34.959349 10136 raft_consensus.cc:2454] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 1.
I20260619 02:34:34.959606 9789 leader_election.cc:302] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9ffe8f71e24cb388303b5fb414f4d2, 8bba1447d9a44e2aad7f415ab8cf00ce; no voters:
I20260619 02:34:34.959782 10211 raft_consensus.cc:2781] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:34.959977 10211 raft_consensus.cc:686] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9ffe8f71e24cb388303b5fb414f4d2, State: Running, Role: LEADER
I20260619 02:34:34.960253 10211 consensus_queue.cc:237] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:34.962538 9697 catalog_manager.cc:5495] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193). New cstate: current_term: 1 leader_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } health_report { overall_health: UNKNOWN } } }
I20260619 02:34:34.978226 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.974642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.7.194.193:46703)
0619 02:34:34.974937 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.974944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.974955 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:34:34.975022 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.975024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.975074 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.975233 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:34.975428 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.975978 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.976731 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.976968 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.977044 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.977271 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:34.977844 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:34.977948 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.978010 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.978055 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":134,"threads_started":1}
I20260619 02:34:34.989187 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.985674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:37553 (local address 127.7.194.193:35785)
0619 02:34:34.985974 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:34.986029 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:34.986139 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:34.986412 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:34.986415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:34.986423 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:34:34.986584 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.986594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.987310 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.987315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.987952 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:34.987960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.988428 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.988431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:34.988535 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.988575 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:34:34.988891 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:34:34.988900 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:34:34.988959 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:34:34.989043 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
I20260619 02:34:34.989173 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:34.985809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:35785 (local address 127.7.194.194:37553)
0619 02:34:34.985947 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:34.985953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:34.986091 (+ 138us) server_negotiation.cc:407] Connection header received
0619 02:34:34.986253 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:34.986256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:34.986314 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:34.986403 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:34.986682 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.987195 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.988076 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:34.988307 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:34.988427 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:34.988711 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:34.988793 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:34.988888 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:34.988978 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 02:34:34.989026 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:34:34.995393 10136 raft_consensus.cc:1261] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Refusing update from remote peer 2b9ffe8f71e24cb388303b5fb414f4d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:34.995757 9995 raft_consensus.cc:1261] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Refusing update from remote peer 2b9ffe8f71e24cb388303b5fb414f4d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:34.995931 10211 consensus_queue.cc:1035] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:34.996246 10257 consensus_queue.cc:1035] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:35.013394 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:35.010097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:51401 (local address 127.7.194.193:46703)
0619 02:34:35.010190 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:35.010194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:35.010213 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:34:35.010255 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:35.010257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:35.010296 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:35.010393 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:35.010699 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:35.011379 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:35.012319 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:35.012552 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:35.012661 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:35.012945 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:35.013019 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:35.013120 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:35.013207 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 02:34:35.013274 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:34:35.013422 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:35.009850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46703 (local address 127.7.194.194:51401)
0619 02:34:35.009995 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:35.010011 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:35.010091 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:35.010425 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:35.010428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:35.010476 (+ 48us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:34:35.010586 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:35.010595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:35.011486 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:35.011489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:35.012210 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:35.012218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:35.012675 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:35.012679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:35.012768 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:35.012804 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:34:35.013126 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:34:35.013132 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:34:35.013182 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:34:35.013293 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 02:34:35.014256 10231 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:34:36.575194 9698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:36.575374 9698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:36.578028 9698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: Sending DeleteTablet for 3 replicas of tablet 1ae6f4b075784dfa99edeffedb07d42c
I20260619 02:34:36.578701 9976 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae6f4b075784dfa99edeffedb07d42c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:36 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:47208
I20260619 02:34:36.578709 9836 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae6f4b075784dfa99edeffedb07d42c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:36 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:33868
I20260619 02:34:36.578778 10116 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae6f4b075784dfa99edeffedb07d42c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:36 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:43388
I20260619 02:34:36.579312 10291 tablet_replica.cc:330] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: stopping tablet replica
I20260619 02:34:36.579463 10291 raft_consensus.cc:2229] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:36.579588 10291 raft_consensus.cc:2258] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:36.579869 10292 tablet_replica.cc:330] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: stopping tablet replica
I20260619 02:34:36.580042 10292 raft_consensus.cc:2229] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:34:36.580380 10292 raft_consensus.cc:2258] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:36.580478 10291 ts_tablet_manager.cc:1905] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:36.581071 10292 ts_tablet_manager.cc:1905] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:36.581452 10290 tablet_replica.cc:330] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: stopping tablet replica
I20260619 02:34:36.581614 10290 raft_consensus.cc:2229] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:36.581749 10290 raft_consensus.cc:2258] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:36.583931 10290 ts_tablet_manager.cc:1905] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:36.584807 10291 ts_tablet_manager.cc:1918] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:34:36.584926 10292 ts_tablet_manager.cc:1918] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:34:36.584993 10291 log.cc:1196] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/1ae6f4b075784dfa99edeffedb07d42c
I20260619 02:34:36.585057 10292 log.cc:1196] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/1ae6f4b075784dfa99edeffedb07d42c
I20260619 02:34:36.585397 10292 ts_tablet_manager.cc:1939] T 1ae6f4b075784dfa99edeffedb07d42c P 2b9ffe8f71e24cb388303b5fb414f4d2: Deleting consensus metadata
I20260619 02:34:36.585768 10291 ts_tablet_manager.cc:1939] T 1ae6f4b075784dfa99edeffedb07d42c P 1442a84ea91b4a5e9763f165397f4519: Deleting consensus metadata
I20260619 02:34:36.586887 10290 ts_tablet_manager.cc:1918] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:34:36.586985 10290 log.cc:1196] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/1ae6f4b075784dfa99edeffedb07d42c
I20260619 02:34:36.587220 9686 catalog_manager.cc:4841] TS 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193:46703): tablet 1ae6f4b075784dfa99edeffedb07d42c (table test-table [id=804ad716b7cc4a54887f38d7722e15f2]) successfully deleted
I20260619 02:34:36.587257 10290 ts_tablet_manager.cc:1939] T 1ae6f4b075784dfa99edeffedb07d42c P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting consensus metadata
I20260619 02:34:36.587518 9684 catalog_manager.cc:4841] TS 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339): tablet 1ae6f4b075784dfa99edeffedb07d42c (table test-table [id=804ad716b7cc4a54887f38d7722e15f2]) successfully deleted
I20260619 02:34:36.588181 9685 catalog_manager.cc:4841] TS 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553): tablet 1ae6f4b075784dfa99edeffedb07d42c (table test-table [id=804ad716b7cc4a54887f38d7722e15f2]) successfully deleted
Jun 19 02:34:36 dist-test-slave-9tkz krb5kdc[9654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:34:36 dist-test-slave-9tkz krb5kdc[9654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:36.614214 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:36.604939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.7.194.254:40879)
0619 02:34:36.605269 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:36.605276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:36.605291 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:34:36.605414 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:36.605417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:36.605461 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:36.605586 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:36.606285 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:36.606876 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:36.607835 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:36.608069 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:36.608152 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:36.610809 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:36.610813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:36.610815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:36.610841 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:36.612364 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:36.612811 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:36.612814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:36.612815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:36.612893 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:36.613185 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:36.613191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:36.613193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:36.613403 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:36.613510 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:36.613814 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 02:34:36.613971 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":157,"threads_started":1}
I20260619 02:34:36.617290 9698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:34:36.617830 9698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:36.622589 9836 tablet_service.cc:1467] Processing CreateTablet for tablet 04374bb46bb9433e9e992ed5316e2ddd (DEFAULT_TABLE table=test-table [id=92ab46125c5549d2a272e0244304f4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:36.622913 9836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04374bb46bb9433e9e992ed5316e2ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:36.622850 10116 tablet_service.cc:1467] Processing CreateTablet for tablet 04374bb46bb9433e9e992ed5316e2ddd (DEFAULT_TABLE table=test-table [id=92ab46125c5549d2a272e0244304f4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:36.623152 10116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04374bb46bb9433e9e992ed5316e2ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:36.625346 10301 tablet_bootstrap.cc:492] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: Bootstrap starting.
I20260619 02:34:36.625635 10302 tablet_bootstrap.cc:492] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: Bootstrap starting.
I20260619 02:34:36.625643 9976 tablet_service.cc:1467] Processing CreateTablet for tablet 04374bb46bb9433e9e992ed5316e2ddd (DEFAULT_TABLE table=test-table [id=92ab46125c5549d2a272e0244304f4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:36.625933 9976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04374bb46bb9433e9e992ed5316e2ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:36.627489 10301 tablet_bootstrap.cc:654] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:36.627740 10302 tablet_bootstrap.cc:654] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:36.629052 10305 tablet_bootstrap.cc:492] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: Bootstrap starting.
I20260619 02:34:36.629087 10302 tablet_bootstrap.cc:492] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: No bootstrap required, opened a new log
I20260619 02:34:36.629192 10302 ts_tablet_manager.cc:1397] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:34:36.629642 10302 raft_consensus.cc:348] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.629781 10302 raft_consensus.cc:374] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:36.629822 10302 raft_consensus.cc:729] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9ffe8f71e24cb388303b5fb414f4d2, State: Initialized, Role: FOLLOWER
I20260619 02:34:36.629868 10305 tablet_bootstrap.cc:654] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:36.629949 10302 consensus_queue.cc:260] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.630188 10301 tablet_bootstrap.cc:492] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: No bootstrap required, opened a new log
I20260619 02:34:36.630229 10302 ts_tablet_manager.cc:1428] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:36.630290 10301 ts_tablet_manager.cc:1397] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:34:36.630874 10301 raft_consensus.cc:348] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.631016 10301 raft_consensus.cc:374] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:36.631058 10301 raft_consensus.cc:729] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1442a84ea91b4a5e9763f165397f4519, State: Initialized, Role: FOLLOWER
I20260619 02:34:36.631084 10305 tablet_bootstrap.cc:492] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: No bootstrap required, opened a new log
I20260619 02:34:36.631163 10305 ts_tablet_manager.cc:1397] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:36.631187 10301 consensus_queue.cc:260] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.631593 10301 ts_tablet_manager.cc:1428] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:36.631577 10305 raft_consensus.cc:348] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.631699 10305 raft_consensus.cc:374] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:36.631737 10305 raft_consensus.cc:729] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bba1447d9a44e2aad7f415ab8cf00ce, State: Initialized, Role: FOLLOWER
I20260619 02:34:36.631875 10305 consensus_queue.cc:260] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.632158 10305 ts_tablet_manager.cc:1428] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:36.867450 10284 raft_consensus.cc:482] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:36.867591 10284 raft_consensus.cc:504] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.868053 10284 leader_election.cc:288] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:36.868517 9995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04374bb46bb9433e9e992ed5316e2ddd" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" is_pre_election: true
I20260619 02:34:36.868568 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04374bb46bb9433e9e992ed5316e2ddd" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519" is_pre_election: true
I20260619 02:34:36.868643 9995 raft_consensus.cc:2454] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 0.
I20260619 02:34:36.868788 10136 raft_consensus.cc:2454] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 0.
I20260619 02:34:36.868989 9789 leader_election.cc:302] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9ffe8f71e24cb388303b5fb414f4d2, 8bba1447d9a44e2aad7f415ab8cf00ce; no voters:
I20260619 02:34:36.869203 10284 raft_consensus.cc:2781] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:36.869256 10284 raft_consensus.cc:482] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:36.869282 10284 raft_consensus.cc:3037] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:36.870355 10284 raft_consensus.cc:504] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.870679 10284 leader_election.cc:288] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 election: Requested vote from peers 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:36.871016 9995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04374bb46bb9433e9e992ed5316e2ddd" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce"
I20260619 02:34:36.871032 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04374bb46bb9433e9e992ed5316e2ddd" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519"
I20260619 02:34:36.871140 9995 raft_consensus.cc:3037] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:36.871140 10136 raft_consensus.cc:3037] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:36.872265 9995 raft_consensus.cc:2454] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 1.
I20260619 02:34:36.872265 10136 raft_consensus.cc:2454] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 1.
I20260619 02:34:36.872547 9789 leader_election.cc:302] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9ffe8f71e24cb388303b5fb414f4d2, 8bba1447d9a44e2aad7f415ab8cf00ce; no voters:
I20260619 02:34:36.872826 10284 raft_consensus.cc:2781] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:36.872946 10284 raft_consensus.cc:686] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9ffe8f71e24cb388303b5fb414f4d2, State: Running, Role: LEADER
I20260619 02:34:36.873088 10284 consensus_queue.cc:237] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.874864 9698 catalog_manager.cc:5495] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193). New cstate: current_term: 1 leader_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } health_report { overall_health: UNKNOWN } } }
I20260619 02:34:36.932185 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:36.929182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.7.194.193:46703)
0619 02:34:36.929436 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:36.929443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:36.929455 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:34:36.929492 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:36.929494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:36.929533 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:36.929620 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:36.929884 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:36.930395 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:36.931206 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:36.931422 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:36.931503 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:36.931798 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:36.931872 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:36.931925 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:36.932024 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 02:34:36.932067 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260619 02:34:36.938771 9995 raft_consensus.cc:1261] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Refusing update from remote peer 2b9ffe8f71e24cb388303b5fb414f4d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:36.939132 10136 raft_consensus.cc:1261] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Refusing update from remote peer 2b9ffe8f71e24cb388303b5fb414f4d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:36.939184 10284 consensus_queue.cc:1035] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:36.939523 10284 consensus_queue.cc:1035] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:36.942031 10272 mvcc.cc:205] Tried to move back new op lower bound from 7298402209537716224 to 7298402209273634816. Current Snapshot: MvccSnapshot[applied={T|T < 7298402209537716224}]
I20260619 02:34:36.972193 9697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:36.972353 9697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:36.974491 9697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: Sending DeleteTablet for 3 replicas of tablet 04374bb46bb9433e9e992ed5316e2ddd
I20260619 02:34:36.975044 9836 tablet_service.cc:1514] Processing DeleteTablet for tablet 04374bb46bb9433e9e992ed5316e2ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:36 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:33868
I20260619 02:34:36.975174 10116 tablet_service.cc:1514] Processing DeleteTablet for tablet 04374bb46bb9433e9e992ed5316e2ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:36 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:43388
I20260619 02:34:36.975171 9976 tablet_service.cc:1514] Processing DeleteTablet for tablet 04374bb46bb9433e9e992ed5316e2ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:36 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:47208
I20260619 02:34:36.975246 10292 tablet_replica.cc:330] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: stopping tablet replica
I20260619 02:34:36.975358 10290 tablet_replica.cc:330] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: stopping tablet replica
I20260619 02:34:36.975358 10292 raft_consensus.cc:2229] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:34:36.975373 10291 tablet_replica.cc:330] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: stopping tablet replica
I20260619 02:34:36.975462 10290 raft_consensus.cc:2229] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:36.975476 10291 raft_consensus.cc:2229] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:36.975572 10290 raft_consensus.cc:2258] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:36.975572 10292 raft_consensus.cc:2258] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:36.975605 10291 raft_consensus.cc:2258] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:36.976156 9697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:34:36.976420 10291 ts_tablet_manager.cc:1905] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:36.976435 10292 ts_tablet_manager.cc:1905] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:34:36.976576 9697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:36.977697 10290 ts_tablet_manager.cc:1905] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:36.978766 10292 ts_tablet_manager.cc:1918] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:34:36.978864 10292 log.cc:1196] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/04374bb46bb9433e9e992ed5316e2ddd
I20260619 02:34:36.979104 10291 ts_tablet_manager.cc:1918] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:34:36.979131 10292 ts_tablet_manager.cc:1939] T 04374bb46bb9433e9e992ed5316e2ddd P 2b9ffe8f71e24cb388303b5fb414f4d2: Deleting consensus metadata
I20260619 02:34:36.979180 10291 log.cc:1196] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/04374bb46bb9433e9e992ed5316e2ddd
I20260619 02:34:36.979465 10291 ts_tablet_manager.cc:1939] T 04374bb46bb9433e9e992ed5316e2ddd P 1442a84ea91b4a5e9763f165397f4519: Deleting consensus metadata
I20260619 02:34:36.980319 10290 ts_tablet_manager.cc:1918] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:34:36.980240 9684 catalog_manager.cc:4841] TS 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339): tablet 04374bb46bb9433e9e992ed5316e2ddd (table test-table [id=92ab46125c5549d2a272e0244304f4d8]) successfully deleted
I20260619 02:34:36.980392 10290 log.cc:1196] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/04374bb46bb9433e9e992ed5316e2ddd
I20260619 02:34:36.980641 10290 ts_tablet_manager.cc:1939] T 04374bb46bb9433e9e992ed5316e2ddd P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting consensus metadata
I20260619 02:34:36.980808 9686 catalog_manager.cc:4841] TS 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193:46703): tablet 04374bb46bb9433e9e992ed5316e2ddd (table test-table [id=92ab46125c5549d2a272e0244304f4d8]) successfully deleted
I20260619 02:34:36.981295 9685 catalog_manager.cc:4841] TS 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553): tablet 04374bb46bb9433e9e992ed5316e2ddd (table test-table [id=92ab46125c5549d2a272e0244304f4d8]) successfully deleted
I20260619 02:34:36.982568 9836 tablet_service.cc:1467] Processing CreateTablet for tablet b2ce1a48f1604f0aa1054b2ba6f484ce (DEFAULT_TABLE table=test-table [id=bec44d73c1974744a30832328036dda3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:36.982681 9976 tablet_service.cc:1467] Processing CreateTablet for tablet b2ce1a48f1604f0aa1054b2ba6f484ce (DEFAULT_TABLE table=test-table [id=bec44d73c1974744a30832328036dda3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:36.982848 9836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ce1a48f1604f0aa1054b2ba6f484ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:36.982931 9976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ce1a48f1604f0aa1054b2ba6f484ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:36.984859 10305 tablet_bootstrap.cc:492] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: Bootstrap starting.
I20260619 02:34:36.984905 10302 tablet_bootstrap.cc:492] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2: Bootstrap starting.
I20260619 02:34:36.985383 10116 tablet_service.cc:1467] Processing CreateTablet for tablet b2ce1a48f1604f0aa1054b2ba6f484ce (DEFAULT_TABLE table=test-table [id=bec44d73c1974744a30832328036dda3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:36.985666 10116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ce1a48f1604f0aa1054b2ba6f484ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:36.985742 10302 tablet_bootstrap.cc:654] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:36.985837 10305 tablet_bootstrap.cc:654] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:36.986999 10302 tablet_bootstrap.cc:492] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2: No bootstrap required, opened a new log
I20260619 02:34:36.987027 10305 tablet_bootstrap.cc:492] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: No bootstrap required, opened a new log
I20260619 02:34:36.987093 10302 ts_tablet_manager.cc:1397] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:36.987124 10305 ts_tablet_manager.cc:1397] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:34:36.987464 10305 raft_consensus.cc:348] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.987488 10302 raft_consensus.cc:348] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.987581 10305 raft_consensus.cc:374] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:36.987589 10302 raft_consensus.cc:374] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:36.987622 10305 raft_consensus.cc:729] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bba1447d9a44e2aad7f415ab8cf00ce, State: Initialized, Role: FOLLOWER
I20260619 02:34:36.987630 10302 raft_consensus.cc:729] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9ffe8f71e24cb388303b5fb414f4d2, State: Initialized, Role: FOLLOWER
I20260619 02:34:36.987754 10302 consensus_queue.cc:260] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.987756 10305 consensus_queue.cc:260] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.988047 10305 ts_tablet_manager.cc:1428] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:36.988060 10302 ts_tablet_manager.cc:1428] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:36.988107 10301 tablet_bootstrap.cc:492] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: Bootstrap starting.
I20260619 02:34:36.989114 10301 tablet_bootstrap.cc:654] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:36.990417 10301 tablet_bootstrap.cc:492] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: No bootstrap required, opened a new log
I20260619 02:34:36.990540 10301 ts_tablet_manager.cc:1397] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:34:36.991008 10301 raft_consensus.cc:348] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.991137 10301 raft_consensus.cc:374] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:36.991173 10301 raft_consensus.cc:729] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1442a84ea91b4a5e9763f165397f4519, State: Initialized, Role: FOLLOWER
I20260619 02:34:36.991297 10301 consensus_queue.cc:260] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:36.991559 10301 ts_tablet_manager.cc:1428] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:37.327395 10280 raft_consensus.cc:482] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:37.327584 10280 raft_consensus.cc:504] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:37.328195 10280 leader_election.cc:288] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:37.328713 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ce1a48f1604f0aa1054b2ba6f484ce" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519" is_pre_election: true
I20260619 02:34:37.328747 9995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ce1a48f1604f0aa1054b2ba6f484ce" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" is_pre_election: true
I20260619 02:34:37.328862 10136 raft_consensus.cc:2454] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 0.
I20260619 02:34:37.328927 9995 raft_consensus.cc:2454] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 0.
I20260619 02:34:37.329221 9788 leader_election.cc:302] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1442a84ea91b4a5e9763f165397f4519, 2b9ffe8f71e24cb388303b5fb414f4d2; no voters:
I20260619 02:34:37.329412 10280 raft_consensus.cc:2781] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:37.329474 10280 raft_consensus.cc:482] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:37.329512 10280 raft_consensus.cc:3037] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:37.330600 10280 raft_consensus.cc:504] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:37.330936 10280 leader_election.cc:288] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 election: Requested vote from peers 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553), 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339)
I20260619 02:34:37.331308 9995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ce1a48f1604f0aa1054b2ba6f484ce" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce"
I20260619 02:34:37.331333 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ce1a48f1604f0aa1054b2ba6f484ce" candidate_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442a84ea91b4a5e9763f165397f4519"
I20260619 02:34:37.331451 10136 raft_consensus.cc:3037] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:37.331451 9995 raft_consensus.cc:3037] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:37.332628 10136 raft_consensus.cc:2454] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 1.
I20260619 02:34:37.332628 9995 raft_consensus.cc:2454] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9ffe8f71e24cb388303b5fb414f4d2 in term 1.
I20260619 02:34:37.332908 9788 leader_election.cc:302] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1442a84ea91b4a5e9763f165397f4519, 2b9ffe8f71e24cb388303b5fb414f4d2; no voters:
I20260619 02:34:37.333134 10280 raft_consensus.cc:2781] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:37.333233 10280 raft_consensus.cc:686] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9ffe8f71e24cb388303b5fb414f4d2, State: Running, Role: LEADER
I20260619 02:34:37.333380 10280 consensus_queue.cc:237] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } }
I20260619 02:34:37.335210 9697 catalog_manager.cc:5495] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193). New cstate: current_term: 1 leader_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9ffe8f71e24cb388303b5fb414f4d2" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 } health_report { overall_health: UNKNOWN } } }
I20260619 02:34:37.362803 10136 raft_consensus.cc:1261] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Refusing update from remote peer 2b9ffe8f71e24cb388303b5fb414f4d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:37.362881 9995 raft_consensus.cc:1261] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Refusing update from remote peer 2b9ffe8f71e24cb388303b5fb414f4d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:37.363188 10280 consensus_queue.cc:1035] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1442a84ea91b4a5e9763f165397f4519" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:37.363493 10284 consensus_queue.cc:1035] T b2ce1a48f1604f0aa1054b2ba6f484ce P 2b9ffe8f71e24cb388303b5fb414f4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bba1447d9a44e2aad7f415ab8cf00ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:34:37.376778 10272 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:34:38.928400 9698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:38.928602 9698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:38.931109 9698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 878acf6dff33426db10cec9ef1ea108c: Sending DeleteTablet for 3 replicas of tablet b2ce1a48f1604f0aa1054b2ba6f484ce
I20260619 02:34:38.931676 9836 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ce1a48f1604f0aa1054b2ba6f484ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:38 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:33868
I20260619 02:34:38.931820 9976 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ce1a48f1604f0aa1054b2ba6f484ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:38 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:47208
I20260619 02:34:38.932210 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 9771
I20260619 02:34:38.932232 10343 tablet_replica.cc:330] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: stopping tablet replica
I20260619 02:34:38.933279 10343 raft_consensus.cc:2229] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:38.933454 10343 raft_consensus.cc:2258] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:38.934230 10343 ts_tablet_manager.cc:1905] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:38.935158 10116 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ce1a48f1604f0aa1054b2ba6f484ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:38 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:43388
I20260619 02:34:38.935498 10344 tablet_replica.cc:330] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: stopping tablet replica
I20260619 02:34:38.935626 10344 raft_consensus.cc:2229] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:38.935767 10344 raft_consensus.cc:2258] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:38.936408 10344 ts_tablet_manager.cc:1905] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:38.937474 10343 ts_tablet_manager.cc:1918] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:34:38.937584 10343 log.cc:1196] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/b2ce1a48f1604f0aa1054b2ba6f484ce
I20260619 02:34:38.937865 10343 ts_tablet_manager.cc:1939] T b2ce1a48f1604f0aa1054b2ba6f484ce P 8bba1447d9a44e2aad7f415ab8cf00ce: Deleting consensus metadata
I20260619 02:34:38.938655 9685 catalog_manager.cc:4841] TS 8bba1447d9a44e2aad7f415ab8cf00ce (127.7.194.194:37553): tablet b2ce1a48f1604f0aa1054b2ba6f484ce (table test-table [id=bec44d73c1974744a30832328036dda3]) successfully deleted
I20260619 02:34:38.939061 10344 ts_tablet_manager.cc:1918] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:34:38.939126 10344 log.cc:1196] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/b2ce1a48f1604f0aa1054b2ba6f484ce
I20260619 02:34:38.939376 10344 ts_tablet_manager.cc:1939] T b2ce1a48f1604f0aa1054b2ba6f484ce P 1442a84ea91b4a5e9763f165397f4519: Deleting consensus metadata
I20260619 02:34:38.940033 9684 catalog_manager.cc:4841] TS 1442a84ea91b4a5e9763f165397f4519 (127.7.194.195:41339): tablet b2ce1a48f1604f0aa1054b2ba6f484ce (table test-table [id=bec44d73c1974744a30832328036dda3]) successfully deleted
W20260619 02:34:38.941282 9686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:46703 (error 108)
I20260619 02:34:38.941628 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 9911
I20260619 02:34:38.943347 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:38.942892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46703 (local address 127.0.0.1:52654)
0619 02:34:38.943175 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:38.943253 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
W20260619 02:34:38.943632 9686 catalog_manager.cc:4584] TS 2b9ffe8f71e24cb388303b5fb414f4d2 (127.7.194.193:46703): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2ce1a48f1604f0aa1054b2ba6f484ce: Network error: Client connection negotiation failed: client connection to 127.7.194.193:46703: connect: Connection refused (error 111)
I20260619 02:34:38.949371 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 10051
I20260619 02:34:38.955250 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 9667
2026-06-19T02:34:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7000 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:39 dist-test-slave-9tkz krb5kdc[10349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:39 dist-test-slave-9tkz krb5kdc[10349](info): set up 2 sockets
Jun 19 02:34:39 dist-test-slave-9tkz krb5kdc[10349](info): commencing operation
krb5kdc: starting...
W20260619 02:34:39.997802 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:40 dist-test-slave-9tkz krb5kdc[10349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:40.125092 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:46767
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:46767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:40.238883 10362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:40.239179 10362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:40.239265 10362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:40.243386 10362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:40.243454 10362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:40.243474 10362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:40.243490 10362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:40.243505 10362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:40.249096 10362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:46767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:40.250341 10362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:40.251508 10362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:40.260941 10370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:40.261003 10362 server_base.cc:1034] running on GCE node
W20260619 02:34:40.260941 10368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:40.260953 10367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:40.261936 10362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:40.263260 10362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:40.264472 10362 hybrid_clock.cc:648] HybridClock initialized: now 1781836480264455 us; error 38 us; skew 500 ppm
Jun 19 02:34:40 dist-test-slave-9tkz krb5kdc[10349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:40.267506 10362 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:40.268570 10362 webserver.cc:460] Webserver started at http://127.7.194.254:37537/ using document root <none> and password file <none>
I20260619 02:34:40.269207 10362 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:40.269311 10362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:40.269631 10362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:40.271960 10362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "b165c8f2df1248c3a9e75431fd1b5dd4"
format_stamp: "Formatted at 2026-06-19 02:34:40 on dist-test-slave-9tkz"
server_key: "bbe65a75e5b12536e9664b8f5048470c"
server_key_iv: "87483762d716d6a71f9ab9d9a46fd9c8"
server_key_version: "encryptionkey@0"
I20260619 02:34:40.272603 10362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "b165c8f2df1248c3a9e75431fd1b5dd4"
format_stamp: "Formatted at 2026-06-19 02:34:40 on dist-test-slave-9tkz"
server_key: "bbe65a75e5b12536e9664b8f5048470c"
server_key_iv: "87483762d716d6a71f9ab9d9a46fd9c8"
server_key_version: "encryptionkey@0"
I20260619 02:34:40.276882 10362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 02:34:40.279774 10377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:40.281131 10362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:34:40.281322 10362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "b165c8f2df1248c3a9e75431fd1b5dd4"
format_stamp: "Formatted at 2026-06-19 02:34:40 on dist-test-slave-9tkz"
server_key: "bbe65a75e5b12536e9664b8f5048470c"
server_key_iv: "87483762d716d6a71f9ab9d9a46fd9c8"
server_key_version: "encryptionkey@0"
I20260619 02:34:40.281459 10362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:40.302076 10362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:40.305662 10362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:40.305915 10362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:40.313885 10362 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:46767
I20260619 02:34:40.313910 10429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:46767 every 8 connection(s)
I20260619 02:34:40.314991 10362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:40.318004 10430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:40.321580 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 10362
I20260619 02:34:40.321744 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:40.322026 7947 external_mini_cluster.cc:1428] Setting key 91cc705fcf9b0f1cc34c61a57a626d26
I20260619 02:34:40.324201 10430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4: Bootstrap starting.
I20260619 02:34:40.326570 10430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:40.327335 10430 log.cc:824] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:40.329223 10430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4: No bootstrap required, opened a new log
Jun 19 02:34:40 dist-test-slave-9tkz krb5kdc[10349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:40.333652 10430 raft_consensus.cc:348] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165c8f2df1248c3a9e75431fd1b5dd4" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46767 } }
I20260619 02:34:40.333931 10430 raft_consensus.cc:374] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:40.334012 10430 raft_consensus.cc:729] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b165c8f2df1248c3a9e75431fd1b5dd4, State: Initialized, Role: FOLLOWER
I20260619 02:34:40.334496 10430 consensus_queue.cc:260] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165c8f2df1248c3a9e75431fd1b5dd4" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46767 } }
I20260619 02:34:40.334648 10430 raft_consensus.cc:388] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:40.334745 10430 raft_consensus.cc:482] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:40.334864 10430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:40.336103 10430 raft_consensus.cc:504] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165c8f2df1248c3a9e75431fd1b5dd4" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46767 } }
I20260619 02:34:40.336524 10430 leader_election.cc:302] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b165c8f2df1248c3a9e75431fd1b5dd4; no voters:
I20260619 02:34:40.336905 10430 leader_election.cc:288] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:40.337218 10435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:40.337479 10435 raft_consensus.cc:686] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [term 1 LEADER]: Becoming Leader. State: Replica: b165c8f2df1248c3a9e75431fd1b5dd4, State: Running, Role: LEADER
I20260619 02:34:40.337906 10435 consensus_queue.cc:237] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165c8f2df1248c3a9e75431fd1b5dd4" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46767 } }
I20260619 02:34:40.338203 10430 sys_catalog.cc:564] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:40.338724 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:40.323573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.7.194.254:46767)
0619 02:34:40.324038 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:40.324055 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:40.324096 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:34:40.324946 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:40.324968 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:40.325330 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:40.325725 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:40.326621 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.327779 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.328714 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.329217 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.329426 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:40.332407 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:40.332412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:40.332426 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:40.332455 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:40.334613 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:40.335186 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:40.335191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:40.335196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:40.335319 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:40.335601 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:40.335603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:40.335604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:40.336170 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:40.336376 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:40.337313 (+ 937us) server_negotiation.cc:299] Negotiation successful
0619 02:34:40.337698 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260619 02:34:40.339648 10436 sys_catalog.cc:455] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b165c8f2df1248c3a9e75431fd1b5dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165c8f2df1248c3a9e75431fd1b5dd4" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46767 } } }
I20260619 02:34:40.339794 10436 sys_catalog.cc:458] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:40.340032 10436 sys_catalog.cc:455] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b165c8f2df1248c3a9e75431fd1b5dd4. Latest consensus state: current_term: 1 leader_uuid: "b165c8f2df1248c3a9e75431fd1b5dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165c8f2df1248c3a9e75431fd1b5dd4" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46767 } } }
I20260619 02:34:40.340126 10436 sys_catalog.cc:458] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:40.341156 10446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:40.343940 10446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:40.349818 10446 catalog_manager.cc:1347] Generated new cluster ID: b51ee2c9b43547a9b1b4bd7e8c969e38
I20260619 02:34:40.349901 10446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:40.373312 10446 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:40.374689 10446 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:40.379281 10446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b165c8f2df1248c3a9e75431fd1b5dd4: Generated new TSK 0
I20260619 02:34:40.380102 10446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:34:40.410563 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46767
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:40.528095 10456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:40.528419 10456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:40.528466 10456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:40.532526 10456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:40.532596 10456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:40.532660 10456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:40.538511 10456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:40.539763 10456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:40.540908 10456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:40.550907 10464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:40.550938 10462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:40.550907 10461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:40.551095 10456 server_base.cc:1034] running on GCE node
I20260619 02:34:40.551527 10456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:40.552227 10456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:40.553418 10456 hybrid_clock.cc:648] HybridClock initialized: now 1781836480553401 us; error 32 us; skew 500 ppm
Jun 19 02:34:40 dist-test-slave-9tkz krb5kdc[10349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:40.559296 10456 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:34:40.560288 10456 webserver.cc:460] Webserver started at http://127.7.194.193:46415/ using document root <none> and password file <none>
I20260619 02:34:40.560842 10456 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:40.560904 10456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:40.561074 10456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:40.563251 10456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "76321765702a49e6b015e38e613e21a5"
format_stamp: "Formatted at 2026-06-19 02:34:40 on dist-test-slave-9tkz"
server_key: "1dbf71c3d6abfb78083aef1ce7dd1fe2"
server_key_iv: "9b5b2c429f28616dad3a1f4ff6a7e92d"
server_key_version: "encryptionkey@0"
I20260619 02:34:40.563798 10456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "76321765702a49e6b015e38e613e21a5"
format_stamp: "Formatted at 2026-06-19 02:34:40 on dist-test-slave-9tkz"
server_key: "1dbf71c3d6abfb78083aef1ce7dd1fe2"
server_key_iv: "9b5b2c429f28616dad3a1f4ff6a7e92d"
server_key_version: "encryptionkey@0"
I20260619 02:34:40.567654 10456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:40.570181 10471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:40.571513 10456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:34:40.571640 10456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "76321765702a49e6b015e38e613e21a5"
format_stamp: "Formatted at 2026-06-19 02:34:40 on dist-test-slave-9tkz"
server_key: "1dbf71c3d6abfb78083aef1ce7dd1fe2"
server_key_iv: "9b5b2c429f28616dad3a1f4ff6a7e92d"
server_key_version: "encryptionkey@0"
I20260619 02:34:40.571760 10456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:40.587602 10456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:40.591727 10456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:40.591975 10456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:40.592602 10456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:40.593470 10456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:40.593518 10456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:40.593613 10456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:40.593647 10456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:40.602162 10456 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:38893
I20260619 02:34:40.602205 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:38893 every 8 connection(s)
I20260619 02:34:40.603192 10456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:40.607822 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 10456
I20260619 02:34:40.607950 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:40.608263 7947 external_mini_cluster.cc:1428] Setting key 37955be9fc81d1522210c536cdf735c8
Jun 19 02:34:40 dist-test-slave-9tkz krb5kdc[10349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:34:40.618304 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:40.605164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:40725 (local address 127.7.194.254:46767)
0619 02:34:40.605308 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:40.605311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:40.606264 (+ 953us) server_negotiation.cc:407] Connection header received
0619 02:34:40.607533 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:40.607536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:40.607584 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:40.607649 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:40.609241 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.609912 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.611083 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.611312 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.611423 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:40.614369 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:40.614372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:40.614374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:40.614401 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:40.615917 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:40.616566 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:40.616568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:40.616569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:40.616642 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:40.617071 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:40.617074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:40.617075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:40.617251 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:40.617329 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:40.618067 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 02:34:40.618176 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:34:40.619378 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:40.605628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:46767 (local address 127.7.194.193:40725)
0619 02:34:40.606103 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:40.606148 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:40.607334 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:40.607895 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:40.607906 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:40.608425 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:40.609092 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:40.609111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.610041 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.610049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:40.610956 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:40.610965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.611462 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.611466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:40.611638 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:40.612260 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:40.612277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:40.614167 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:40.616044 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:40.616053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:40.616073 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:40.616459 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:40.616735 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:40.616739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:40.616741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:40.616960 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:40.617367 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:40.617375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:40.617768 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 02:34:40.618013 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 02:34:40.618394 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:34:40.620921 10585 heartbeater.cc:347] Connected to a master server at 127.7.194.254:46767
I20260619 02:34:40.621186 10585 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:40.621765 10585 heartbeater.cc:511] Master 127.7.194.254:46767 requested a full tablet report, sending...
I20260619 02:34:40.623371 10394 ts_manager.cc:194] Registered new tserver with Master: 76321765702a49e6b015e38e613e21a5 (127.7.194.193:38893)
I20260619 02:34:40.625056 10394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:40725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:34:40.638365 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46767
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:40.755177 10590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:40.755455 10590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:40.755499 10590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:40.759531 10590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:40.759593 10590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:40.759654 10590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:40.765110 10590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:40.766216 10590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:40.767235 10590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:40.777590 10595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:40.777590 10598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:40.777539 10596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:40.777644 10590 server_base.cc:1034] running on GCE node
I20260619 02:34:40.778187 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:40.778899 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:40.780066 10590 hybrid_clock.cc:648] HybridClock initialized: now 1781836480780033 us; error 44 us; skew 500 ppm
Jun 19 02:34:40 dist-test-slave-9tkz krb5kdc[10349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:40.782971 10590 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:34:40.783910 10590 webserver.cc:460] Webserver started at http://127.7.194.194:45171/ using document root <none> and password file <none>
I20260619 02:34:40.784487 10590 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:40.784579 10590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:40.784859 10590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:40.787331 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "626deabd91b44761b7a270ffe2aaafef"
format_stamp: "Formatted at 2026-06-19 02:34:40 on dist-test-slave-9tkz"
server_key: "2619380f978d6f190633ca9f2536a657"
server_key_iv: "f63e243755026180be05e5d185c296bd"
server_key_version: "encryptionkey@0"
I20260619 02:34:40.788018 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "626deabd91b44761b7a270ffe2aaafef"
format_stamp: "Formatted at 2026-06-19 02:34:40 on dist-test-slave-9tkz"
server_key: "2619380f978d6f190633ca9f2536a657"
server_key_iv: "f63e243755026180be05e5d185c296bd"
server_key_version: "encryptionkey@0"
I20260619 02:34:40.792117 10590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:40.794817 10605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:40.796109 10590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:40.796268 10590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "626deabd91b44761b7a270ffe2aaafef"
format_stamp: "Formatted at 2026-06-19 02:34:40 on dist-test-slave-9tkz"
server_key: "2619380f978d6f190633ca9f2536a657"
server_key_iv: "f63e243755026180be05e5d185c296bd"
server_key_version: "encryptionkey@0"
I20260619 02:34:40.796389 10590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:40.812042 10590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:40.815078 10590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:40.815302 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:40.815927 10590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:40.816802 10590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:40.816869 10590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:40.816944 10590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:40.816996 10590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:40.826114 10590 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:37561
I20260619 02:34:40.826135 10718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:37561 every 8 connection(s)
I20260619 02:34:40.827104 10590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:40.835798 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 10590
I20260619 02:34:40.835932 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:40.836270 7947 external_mini_cluster.cc:1428] Setting key 0c331225bda745332c19e0b50f1c8c7d
Jun 19 02:34:40 dist-test-slave-9tkz krb5kdc[10349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:40.842571 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:40.829094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:59851 (local address 127.7.194.254:46767)
0619 02:34:40.829244 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:40.829252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:40.830348 (+ 1096us) server_negotiation.cc:407] Connection header received
0619 02:34:40.831256 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:40.831258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:40.831307 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:40.831383 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:40.832859 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.833412 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.834774 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.835003 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.835091 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:40.838340 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:40.838343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:40.838345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:40.838393 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:40.840111 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:40.840820 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:40.840823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:40.840825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:40.840905 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:40.841313 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:40.841315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:40.841316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:40.841494 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:40.841640 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:40.842306 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 02:34:40.842445 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:34:40.843946 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:40.829415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:46767 (local address 127.7.194.194:59851)
0619 02:34:40.830176 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:40.830222 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:40.831044 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:40.831776 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:40.831784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:40.832163 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:40.832673 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:40.832688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.833526 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.833528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:40.834646 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:40.834656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:40.835163 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:40.835167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:40.835324 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:40.836075 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:40.836111 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:40.838136 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:40.840235 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:40.840245 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:40.840266 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:40.840696 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:40.840999 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:40.841005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:40.841009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:40.841204 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:40.841616 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:40.841627 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:40.841989 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 02:34:40.842228 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 02:34:40.842651 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":187,"threads_started":1}
I20260619 02:34:40.845378 10719 heartbeater.cc:347] Connected to a master server at 127.7.194.254:46767
I20260619 02:34:40.845713 10719 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:40.846259 10719 heartbeater.cc:511] Master 127.7.194.254:46767 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:34:40.847431 10394 ts_manager.cc:194] Registered new tserver with Master: 626deabd91b44761b7a270ffe2aaafef (127.7.194.194:37561)
I20260619 02:34:40.848049 10394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:59851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:34:40.864794 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46767
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:40.979753 10724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:40.980048 10724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:40.980129 10724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:40.984297 10724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:40.984369 10724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:40.984431 10724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:40.990077 10724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:40.991266 10724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:40.992301 10724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:34:41.002342 10724 server_base.cc:1034] running on GCE node
W20260619 02:34:41.002279 10730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:41.002269 10729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:41.002261 10732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:41.003077 10724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:41.003723 10724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:41.004892 10724 hybrid_clock.cc:648] HybridClock initialized: now 1781836481004854 us; error 50 us; skew 500 ppm
Jun 19 02:34:41 dist-test-slave-9tkz krb5kdc[10349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:41.007756 10724 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:34:41.008694 10724 webserver.cc:460] Webserver started at http://127.7.194.195:45431/ using document root <none> and password file <none>
I20260619 02:34:41.009305 10724 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:41.009399 10724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:41.009691 10724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:41.011828 10724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "3d6a91c2163a43139b06960675186da8"
format_stamp: "Formatted at 2026-06-19 02:34:41 on dist-test-slave-9tkz"
server_key: "aca904eb9036c27b8a1e77f413ec8eaa"
server_key_iv: "d3d5e490f9cd40334ff40935d682c439"
server_key_version: "encryptionkey@0"
I20260619 02:34:41.012423 10724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "3d6a91c2163a43139b06960675186da8"
format_stamp: "Formatted at 2026-06-19 02:34:41 on dist-test-slave-9tkz"
server_key: "aca904eb9036c27b8a1e77f413ec8eaa"
server_key_iv: "d3d5e490f9cd40334ff40935d682c439"
server_key_version: "encryptionkey@0"
I20260619 02:34:41.016364 10724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 02:34:41.018908 10739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:41.020218 10724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:41.020376 10724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "3d6a91c2163a43139b06960675186da8"
format_stamp: "Formatted at 2026-06-19 02:34:41 on dist-test-slave-9tkz"
server_key: "aca904eb9036c27b8a1e77f413ec8eaa"
server_key_iv: "d3d5e490f9cd40334ff40935d682c439"
server_key_version: "encryptionkey@0"
I20260619 02:34:41.020503 10724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:41.044325 10724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:41.047477 10724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:41.047703 10724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:41.048352 10724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:41.049264 10724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:41.049338 10724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:41.049409 10724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:41.049458 10724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:41.058780 10724 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:38833
I20260619 02:34:41.058799 10852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:38833 every 8 connection(s)
I20260619 02:34:41.059768 10724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:41.061637 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 10724
I20260619 02:34:41.061743 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:41.062037 7947 external_mini_cluster.cc:1428] Setting key 86832ec1ba1ce851a0345dde39c6a480
Jun 19 02:34:41 dist-test-slave-9tkz krb5kdc[10349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:41.074563 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:41.061792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:58437 (local address 127.7.194.254:46767)
0619 02:34:41.061939 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:41.061943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:41.063092 (+ 1149us) server_negotiation.cc:407] Connection header received
0619 02:34:41.064796 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:41.064799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:41.064853 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:41.064930 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:41.066245 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:41.066800 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:41.067972 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:41.068208 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:41.068323 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:41.070943 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:41.070946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:41.070948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:41.070997 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:41.072543 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:41.073057 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:41.073060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:41.073062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:41.073142 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:41.073482 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:41.073485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:41.073487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:41.073746 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:41.073830 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:41.074323 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 02:34:41.074434 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:34:41.075634 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:41.062323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:46767 (local address 127.7.194.195:58437)
0619 02:34:41.062937 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:41.063005 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:41.064575 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:41.065113 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:41.065121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:41.065484 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:41.066096 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:41.066111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:41.066930 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:41.066934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:41.067843 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:41.067852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:41.068360 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:41.068363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:41.068549 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:41.069149 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:41.069166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:41.070745 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:41.072652 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:41.072656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:41.072670 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:41.072952 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:41.073221 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:41.073223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:41.073225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:41.073364 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:41.073848 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:41.073853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:41.074107 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 02:34:41.074261 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 02:34:41.074600 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":161,"threads_started":1}
I20260619 02:34:41.076764 10853 heartbeater.cc:347] Connected to a master server at 127.7.194.254:46767
I20260619 02:34:41.076995 10853 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:41.077468 10853 heartbeater.cc:511] Master 127.7.194.254:46767 requested a full tablet report, sending...
I20260619 02:34:41.078554 10394 ts_manager.cc:194] Registered new tserver with Master: 3d6a91c2163a43139b06960675186da8 (127.7.194.195:38833)
I20260619 02:34:41.079195 10394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:58437
I20260619 02:34:41.088826 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:34:41.103082 10433 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:34:41.101322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.7.194.254:46767)
0619 02:34:41.101593 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:41.101599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:41.101707 (+ 108us) server_negotiation.cc:407] Connection header received
0619 02:34:41.101925 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:41.101929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:41.101986 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:41.102080 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:41.102903 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41342: BlockingRecv error: recv got EOF from 127.0.0.1:41342 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
I20260619 02:34:41.104462 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 10456
I20260619 02:34:41.110148 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 10590
I20260619 02:34:41.115630 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 10724
I20260619 02:34:41.121146 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 10362
2026-06-19T02:34:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2164 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:41 dist-test-slave-9tkz krb5kdc[10865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:41 dist-test-slave-9tkz krb5kdc[10865](info): set up 2 sockets
Jun 19 02:34:41 dist-test-slave-9tkz krb5kdc[10865](info): commencing operation
krb5kdc: starting...
W20260619 02:34:42.165666 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:42 dist-test-slave-9tkz krb5kdc[10865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:42.288403 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:43621
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:42.402088 10878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:42.402387 10878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:42.402428 10878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:42.406761 10878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:42.406844 10878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:42.406865 10878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:42.406883 10878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:42.406899 10878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:42.412607 10878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:43621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:42.413832 10878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:42.414887 10878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:42.423687 10886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:42.423674 10883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:42.423724 10878 server_base.cc:1034] running on GCE node
W20260619 02:34:42.423674 10884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:42.424638 10878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:42.425894 10878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:42.427094 10878 hybrid_clock.cc:648] HybridClock initialized: now 1781836482427061 us; error 48 us; skew 500 ppm
Jun 19 02:34:42 dist-test-slave-9tkz krb5kdc[10865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836482, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:42.429963 10878 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:42.430919 10878 webserver.cc:460] Webserver started at http://127.7.194.254:46829/ using document root <none> and password file <none>
I20260619 02:34:42.431519 10878 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:42.431617 10878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:42.431862 10878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:42.434160 10878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "55c3c566f997401490568b8a1450c4df"
format_stamp: "Formatted at 2026-06-19 02:34:42 on dist-test-slave-9tkz"
server_key: "e4bd4d4cb1607298d95ea009b9b87735"
server_key_iv: "fb1943113043d94079539d6f957db775"
server_key_version: "encryptionkey@0"
I20260619 02:34:42.434794 10878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "55c3c566f997401490568b8a1450c4df"
format_stamp: "Formatted at 2026-06-19 02:34:42 on dist-test-slave-9tkz"
server_key: "e4bd4d4cb1607298d95ea009b9b87735"
server_key_iv: "fb1943113043d94079539d6f957db775"
server_key_version: "encryptionkey@0"
I20260619 02:34:42.438922 10878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:42.441519 10893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:42.442984 10878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:42.443133 10878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "55c3c566f997401490568b8a1450c4df"
format_stamp: "Formatted at 2026-06-19 02:34:42 on dist-test-slave-9tkz"
server_key: "e4bd4d4cb1607298d95ea009b9b87735"
server_key_iv: "fb1943113043d94079539d6f957db775"
server_key_version: "encryptionkey@0"
I20260619 02:34:42.443259 10878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:42.455600 10878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:42.458884 10878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:42.459100 10878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:42.466701 10878 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:43621
I20260619 02:34:42.467048 10945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:43621 every 8 connection(s)
I20260619 02:34:42.467772 10878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:42.470819 10946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:42.474907 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 10878
I20260619 02:34:42.475008 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:42.475322 7947 external_mini_cluster.cc:1428] Setting key ce9767669b4a58b2f3748a2393925d1f
I20260619 02:34:42.477152 10946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df: Bootstrap starting.
I20260619 02:34:42.479830 10946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:42.480600 10946 log.cc:824] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:42.482956 10946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df: No bootstrap required, opened a new log
Jun 19 02:34:42 dist-test-slave-9tkz krb5kdc[10865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:42.486589 10946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c3c566f997401490568b8a1450c4df" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43621 } }
I20260619 02:34:42.486873 10946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:42.486955 10946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c3c566f997401490568b8a1450c4df, State: Initialized, Role: FOLLOWER
I20260619 02:34:42.487429 10946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c3c566f997401490568b8a1450c4df" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43621 } }
I20260619 02:34:42.487592 10946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:42.487676 10946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:42.487802 10946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:42.488757 10946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c3c566f997401490568b8a1450c4df" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43621 } }
I20260619 02:34:42.489180 10946 leader_election.cc:302] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55c3c566f997401490568b8a1450c4df; no voters:
I20260619 02:34:42.489596 10946 leader_election.cc:288] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:42.489727 10951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:42.490002 10951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [term 1 LEADER]: Becoming Leader. State: Replica: 55c3c566f997401490568b8a1450c4df, State: Running, Role: LEADER
I20260619 02:34:42.490419 10951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c3c566f997401490568b8a1450c4df" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43621 } }
I20260619 02:34:42.491088 10946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:42.492101 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:42.476731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.7.194.254:43621)
0619 02:34:42.477167 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:42.477183 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:42.477227 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:34:42.477983 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:42.478003 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:42.478363 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:42.478658 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:42.479608 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:42.480681 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:42.481825 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:42.482335 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:42.482464 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:42.485453 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:42.485459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:42.485475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:42.485506 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:42.488042 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:42.488521 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:42.488527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:42.488534 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:42.488662 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:42.488987 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:42.488991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:42.488992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:42.489596 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:42.490469 (+ 873us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:42.490629 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 02:34:42.490958 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":150,"threads_started":1}
I20260619 02:34:42.492094 10952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55c3c566f997401490568b8a1450c4df. Latest consensus state: current_term: 1 leader_uuid: "55c3c566f997401490568b8a1450c4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c3c566f997401490568b8a1450c4df" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43621 } } }
I20260619 02:34:42.492377 10952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:42.492688 10953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55c3c566f997401490568b8a1450c4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c3c566f997401490568b8a1450c4df" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43621 } } }
I20260619 02:34:42.492794 10953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:42.492856 10959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:42.495664 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:34:42.496279 10959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:42.502912 10959 catalog_manager.cc:1347] Generated new cluster ID: 60a736a9612b4fbda57ee874d392a7f3
I20260619 02:34:42.503002 10959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:42.521456 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:42.515503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.7.194.254:43621)
0619 02:34:42.515644 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:42.515649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:42.516012 (+ 363us) server_negotiation.cc:407] Connection header received
0619 02:34:42.516128 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:42.516131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:42.516188 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:42.516283 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:42.516852 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:42.517689 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:42.518841 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:42.519215 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:42.519321 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:42.520275 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:42.520279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:42.520282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:42.520973 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:42.521010 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:42.521030 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:42.521121 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:42.521223 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 02:34:42.521293 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:34:42.533504 10959 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:42.534719 10959 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:42.539041 10959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55c3c566f997401490568b8a1450c4df: Generated new TSK 0
I20260619 02:34:42.539918 10959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:34:42.554214 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 10878
2026-06-19T02:34:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1430 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:42 dist-test-slave-9tkz krb5kdc[10979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:42 dist-test-slave-9tkz krb5kdc[10979](info): set up 2 sockets
Jun 19 02:34:42 dist-test-slave-9tkz krb5kdc[10979](info): commencing operation
krb5kdc: starting...
W20260619 02:34:43.597821 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:43 dist-test-slave-9tkz krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:43.721333 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:34799
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:34799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:43.838466 10992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:43.838753 10992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:43.838801 10992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:43.843015 10992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:43.843086 10992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:43.843106 10992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:43.843123 10992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:43.843139 10992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:43.848517 10992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:34799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:34799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:43.849618 10992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:43.850668 10992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:43.859560 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:43.859614 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:43.859596 10992 server_base.cc:1034] running on GCE node
W20260619 02:34:43.859563 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:43.860468 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:43.861630 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:43.862802 10992 hybrid_clock.cc:648] HybridClock initialized: now 1781836483862792 us; error 27 us; skew 500 ppm
Jun 19 02:34:43 dist-test-slave-9tkz krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836483, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:43.865646 10992 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:43.866540 10992 webserver.cc:460] Webserver started at http://127.7.194.254:33035/ using document root <none> and password file <none>
I20260619 02:34:43.867122 10992 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:43.867192 10992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:43.867383 10992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:43.869462 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "c11db23ba7144173938e5db540005d82"
format_stamp: "Formatted at 2026-06-19 02:34:43 on dist-test-slave-9tkz"
server_key: "a145935db647d6af549f0d3830286417"
server_key_iv: "8a7554ed07540d7a09dac745778c43ed"
server_key_version: "encryptionkey@0"
I20260619 02:34:43.870050 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "c11db23ba7144173938e5db540005d82"
format_stamp: "Formatted at 2026-06-19 02:34:43 on dist-test-slave-9tkz"
server_key: "a145935db647d6af549f0d3830286417"
server_key_iv: "8a7554ed07540d7a09dac745778c43ed"
server_key_version: "encryptionkey@0"
I20260619 02:34:43.873888 10992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:43.876307 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:43.877707 10992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:34:43.877844 10992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "c11db23ba7144173938e5db540005d82"
format_stamp: "Formatted at 2026-06-19 02:34:43 on dist-test-slave-9tkz"
server_key: "a145935db647d6af549f0d3830286417"
server_key_iv: "8a7554ed07540d7a09dac745778c43ed"
server_key_version: "encryptionkey@0"
I20260619 02:34:43.877960 10992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:43.903425 10992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:43.906713 10992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:43.906939 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:43.914237 10992 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:34799
I20260619 02:34:43.914520 11059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:34799 every 8 connection(s)
I20260619 02:34:43.915307 10992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:43.918052 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 10992
I20260619 02:34:43.918176 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:43.918345 11060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:43.918507 7947 external_mini_cluster.cc:1428] Setting key 8b6fb9779c6dfc857eb527121a024e3d
I20260619 02:34:43.924862 11060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82: Bootstrap starting.
Jun 19 02:34:43 dist-test-slave-9tkz krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:43.927300 11060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:43.928164 11060 log.cc:824] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:43.930440 11060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82: No bootstrap required, opened a new log
I20260619 02:34:43.932803 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:43.919944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.7.194.254:34799)
0619 02:34:43.920364 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:43.920380 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:43.920415 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:34:43.921145 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:43.921166 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:43.921502 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:43.921930 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:43.922735 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:43.923750 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:43.924610 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:43.925162 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:43.925329 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:43.927971 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:43.927976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:43.927989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:43.928015 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:43.929940 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:43.930465 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:43.930469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:43.930474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:43.930625 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:43.930909 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:43.930911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:43.930912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:43.931293 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:43.931440 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:43.931730 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 02:34:43.932038 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":126,"threads_started":1}
I20260619 02:34:43.934561 11060 raft_consensus.cc:348] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11db23ba7144173938e5db540005d82" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34799 } }
I20260619 02:34:43.934837 11060 raft_consensus.cc:374] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:43.934897 11060 raft_consensus.cc:729] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11db23ba7144173938e5db540005d82, State: Initialized, Role: FOLLOWER
I20260619 02:34:43.935446 11060 consensus_queue.cc:260] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11db23ba7144173938e5db540005d82" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34799 } }
I20260619 02:34:43.935645 11060 raft_consensus.cc:388] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:43.935732 11060 raft_consensus.cc:482] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:43.935854 11060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:43.936915 11060 raft_consensus.cc:504] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11db23ba7144173938e5db540005d82" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34799 } }
I20260619 02:34:43.937203 11060 leader_election.cc:302] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c11db23ba7144173938e5db540005d82; no voters:
I20260619 02:34:43.937489 11060 leader_election.cc:288] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:43.937697 11065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:43.938078 11065 raft_consensus.cc:686] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [term 1 LEADER]: Becoming Leader. State: Replica: c11db23ba7144173938e5db540005d82, State: Running, Role: LEADER
I20260619 02:34:43.938493 11065 consensus_queue.cc:237] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11db23ba7144173938e5db540005d82" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34799 } }
I20260619 02:34:43.938862 11060 sys_catalog.cc:564] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:43.940184 11066 sys_catalog.cc:455] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c11db23ba7144173938e5db540005d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11db23ba7144173938e5db540005d82" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34799 } } }
I20260619 02:34:43.940320 11066 sys_catalog.cc:458] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:43.940719 11074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:43.941118 11067 sys_catalog.cc:455] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c11db23ba7144173938e5db540005d82. Latest consensus state: current_term: 1 leader_uuid: "c11db23ba7144173938e5db540005d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11db23ba7144173938e5db540005d82" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34799 } } }
I20260619 02:34:43.941291 11067 sys_catalog.cc:458] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:43.943732 11074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:43.949033 11074 catalog_manager.cc:1347] Generated new cluster ID: 770645831480471f93da3a54a4e69321
I20260619 02:34:43.949110 11074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:43.977212 11074 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:43.978313 11074 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:43.984695 11074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c11db23ba7144173938e5db540005d82: Generated new TSK 0
I20260619 02:34:43.985491 11074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:34:44.014952 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34799
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:44.132712 11086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:44.133003 11086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:44.133050 11086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:44.137184 11086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:44.137259 11086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:44.137319 11086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:44.143231 11086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:44.144524 11086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:44.145666 11086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:44.155932 11091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:44.155932 11094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:44.156024 11086 server_base.cc:1034] running on GCE node
W20260619 02:34:44.155932 11092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:44.156633 11086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:44.157294 11086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:44.158486 11086 hybrid_clock.cc:648] HybridClock initialized: now 1781836484158462 us; error 37 us; skew 500 ppm
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836484, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:44.161360 11086 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:34:44.162303 11086 webserver.cc:460] Webserver started at http://127.7.194.193:42273/ using document root <none> and password file <none>
I20260619 02:34:44.162896 11086 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:44.162990 11086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:44.163219 11086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:44.165366 11086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "9b585d9259c24696a004136e16a8c9ec"
format_stamp: "Formatted at 2026-06-19 02:34:44 on dist-test-slave-9tkz"
server_key: "b1b21af1a1ece88ebac30e3414f80052"
server_key_iv: "aac6e6818b30abeb338fda389a29fb60"
server_key_version: "encryptionkey@0"
I20260619 02:34:44.166007 11086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "9b585d9259c24696a004136e16a8c9ec"
format_stamp: "Formatted at 2026-06-19 02:34:44 on dist-test-slave-9tkz"
server_key: "b1b21af1a1ece88ebac30e3414f80052"
server_key_iv: "aac6e6818b30abeb338fda389a29fb60"
server_key_version: "encryptionkey@0"
I20260619 02:34:44.169958 11086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:44.172524 11101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:44.174187 11086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:44.174345 11086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "9b585d9259c24696a004136e16a8c9ec"
format_stamp: "Formatted at 2026-06-19 02:34:44 on dist-test-slave-9tkz"
server_key: "b1b21af1a1ece88ebac30e3414f80052"
server_key_iv: "aac6e6818b30abeb338fda389a29fb60"
server_key_version: "encryptionkey@0"
I20260619 02:34:44.174474 11086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:44.190838 11086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:44.194427 11086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:44.194646 11086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:44.195295 11086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:44.196184 11086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:44.196260 11086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:44.196338 11086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:44.196393 11086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:44.206056 11086 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:39267
I20260619 02:34:44.206080 11214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:39267 every 8 connection(s)
I20260619 02:34:44.207101 11086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:44.212284 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 11086
I20260619 02:34:44.212433 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:44.212713 7947 external_mini_cluster.cc:1428] Setting key 9b9830db8bc6c2a490e9241e3ed22a78
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836484, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:34:44.222325 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:44.209144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:46103 (local address 127.7.194.254:34799)
0619 02:34:44.209324 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:44.209329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:44.210258 (+ 929us) server_negotiation.cc:407] Connection header received
0619 02:34:44.211700 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:44.211704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:44.211767 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:44.211848 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:44.213222 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.213868 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.215177 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.215411 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.215526 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:44.218461 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:44.218464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:44.218466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:44.218490 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:44.220193 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:44.220738 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:44.220742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:44.220743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:44.220837 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:44.221172 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:44.221176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:44.221178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:44.221369 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:44.221474 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:44.222023 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 02:34:44.222153 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:34:44.223201 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:44.209588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:34799 (local address 127.7.194.193:46103)
0619 02:34:44.210089 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:44.210135 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:44.211465 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:44.212024 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:44.212032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:44.212411 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:44.213064 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:44.213079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.213999 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.214003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:44.215043 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:44.215053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.215568 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.215573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:44.215770 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:44.216425 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:44.216441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:44.218273 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:44.220339 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:44.220345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:44.220362 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:44.220628 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:44.220933 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:44.220937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:44.220939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:44.221076 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:44.221484 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:44.221493 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:44.221793 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 02:34:44.221984 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 02:34:44.222319 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":135,"threads_started":1}
I20260619 02:34:44.224592 11215 heartbeater.cc:347] Connected to a master server at 127.7.194.254:34799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:34:44.224848 11215 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:44.225356 11215 heartbeater.cc:511] Master 127.7.194.254:34799 requested a full tablet report, sending...
I20260619 02:34:44.226979 11024 ts_manager.cc:194] Registered new tserver with Master: 9b585d9259c24696a004136e16a8c9ec (127.7.194.193:39267)
I20260619 02:34:44.228147 11024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:46103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:34:44.241664 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34799
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:44.357059 11220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:44.357476 11220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:44.357612 11220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:44.361933 11220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:44.362057 11220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:44.362164 11220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:44.367898 11220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:44.369267 11220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:44.370699 11220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:44.381572 11226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:44.381664 11220 server_base.cc:1034] running on GCE node
W20260619 02:34:44.381901 11225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:44.381908 11228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:44.382448 11220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:44.383203 11220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:44.384368 11220 hybrid_clock.cc:648] HybridClock initialized: now 1781836484384347 us; error 37 us; skew 500 ppm
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836484, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:44.387326 11220 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:34:44.388324 11220 webserver.cc:460] Webserver started at http://127.7.194.194:39699/ using document root <none> and password file <none>
I20260619 02:34:44.388883 11220 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:44.388943 11220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:44.389113 11220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:44.391321 11220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "79d386049ff14a359b9f3a877880c5bf"
format_stamp: "Formatted at 2026-06-19 02:34:44 on dist-test-slave-9tkz"
server_key: "b5bdcd127888db1b13509d9ae3065416"
server_key_iv: "b48a6d64c7ade3f8c0a70e212cc1615c"
server_key_version: "encryptionkey@0"
I20260619 02:34:44.391894 11220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "79d386049ff14a359b9f3a877880c5bf"
format_stamp: "Formatted at 2026-06-19 02:34:44 on dist-test-slave-9tkz"
server_key: "b5bdcd127888db1b13509d9ae3065416"
server_key_iv: "b48a6d64c7ade3f8c0a70e212cc1615c"
server_key_version: "encryptionkey@0"
I20260619 02:34:44.395797 11220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:44.398579 11235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:44.400120 11220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:44.400233 11220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "79d386049ff14a359b9f3a877880c5bf"
format_stamp: "Formatted at 2026-06-19 02:34:44 on dist-test-slave-9tkz"
server_key: "b5bdcd127888db1b13509d9ae3065416"
server_key_iv: "b48a6d64c7ade3f8c0a70e212cc1615c"
server_key_version: "encryptionkey@0"
I20260619 02:34:44.400344 11220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:44.416435 11220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:44.420296 11220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:44.420476 11220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:44.421083 11220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:44.422027 11220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:44.422076 11220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:44.422119 11220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:44.422134 11220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:44.431660 11220 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:39511
I20260619 02:34:44.431689 11348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:39511 every 8 connection(s)
I20260619 02:34:44.432622 11220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:44.438433 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 11220
I20260619 02:34:44.438530 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:44.438927 7947 external_mini_cluster.cc:1428] Setting key 9f97e73852a2f131397ab7b0c92c7e3c
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836484, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:44.448755 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:44.434647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:44635 (local address 127.7.194.254:34799)
0619 02:34:44.434834 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:44.434839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:44.435744 (+ 905us) server_negotiation.cc:407] Connection header received
0619 02:34:44.437182 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:44.437196 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:44.437257 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:44.437338 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:44.438760 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.439557 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.441103 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.441356 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.441478 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:44.444350 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:44.444355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:44.444362 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:44.444399 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:44.446305 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:44.446906 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:44.446910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:44.446912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:44.447013 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:44.447385 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:44.447390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:44.447393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:44.447679 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:44.447791 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:44.448363 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 02:34:44.448527 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:34:44.449654 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:44.435053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:34799 (local address 127.7.194.194:44635)
0619 02:34:44.435568 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:44.435615 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:44.436950 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:44.437523 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:44.437531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:44.437998 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:44.438570 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:44.438591 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.439742 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.439746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:44.440981 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:44.440991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.441525 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.441529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:44.441760 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:44.442503 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:44.442521 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:44.444105 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:44.446451 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:44.446456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:44.446473 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:44.446788 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:44.447113 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:44.447116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:44.447117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:44.447284 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:44.447798 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:44.447804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:44.448085 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 02:34:44.448273 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 02:34:44.448573 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":121,"threads_started":1}
I20260619 02:34:44.450956 11349 heartbeater.cc:347] Connected to a master server at 127.7.194.254:34799
I20260619 02:34:44.451222 11349 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:44.451746 11349 heartbeater.cc:511] Master 127.7.194.254:34799 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:34:44.452876 11024 ts_manager.cc:194] Registered new tserver with Master: 79d386049ff14a359b9f3a877880c5bf (127.7.194.194:39511)
I20260619 02:34:44.453483 11024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:44635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:34:44.468616 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34799
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:44.586912 11354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:44.587332 11354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:44.587420 11354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:44.593658 11354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:44.593732 11354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:44.593840 11354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:44.599705 11354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:44.600944 11354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:44.602157 11354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:44.612967 11362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:44.612946 11359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:44.613158 11354 server_base.cc:1034] running on GCE node
W20260619 02:34:44.612946 11360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:44.613808 11354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:44.614588 11354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:44.615809 11354 hybrid_clock.cc:648] HybridClock initialized: now 1781836484615787 us; error 35 us; skew 500 ppm
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836484, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:44.618957 11354 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:34:44.619985 11354 webserver.cc:460] Webserver started at http://127.7.194.195:43135/ using document root <none> and password file <none>
I20260619 02:34:44.620584 11354 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:44.620679 11354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:44.620924 11354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:44.623153 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "6899cdb9809c43a4a04aa9b13eba3c19"
format_stamp: "Formatted at 2026-06-19 02:34:44 on dist-test-slave-9tkz"
server_key: "7f8b1d77f1d35f22a727dd95edb40f4e"
server_key_iv: "3f02193d082dda344240650603235c53"
server_key_version: "encryptionkey@0"
I20260619 02:34:44.623764 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "6899cdb9809c43a4a04aa9b13eba3c19"
format_stamp: "Formatted at 2026-06-19 02:34:44 on dist-test-slave-9tkz"
server_key: "7f8b1d77f1d35f22a727dd95edb40f4e"
server_key_iv: "3f02193d082dda344240650603235c53"
server_key_version: "encryptionkey@0"
I20260619 02:34:44.627949 11354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:44.630596 11369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:44.631942 11354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:44.632090 11354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "6899cdb9809c43a4a04aa9b13eba3c19"
format_stamp: "Formatted at 2026-06-19 02:34:44 on dist-test-slave-9tkz"
server_key: "7f8b1d77f1d35f22a727dd95edb40f4e"
server_key_iv: "3f02193d082dda344240650603235c53"
server_key_version: "encryptionkey@0"
I20260619 02:34:44.632215 11354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:44.649215 11354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:44.652464 11354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:44.652690 11354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:44.653328 11354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:44.654271 11354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:44.654343 11354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:44.654404 11354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:44.654453 11354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:44.663753 11354 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:44243
I20260619 02:34:44.663770 11482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:44243 every 8 connection(s)
I20260619 02:34:44.664721 11354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:44.665372 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 11354
I20260619 02:34:44.665505 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:44.665838 7947 external_mini_cluster.cc:1428] Setting key 55a1375ddbf975088d0df7bfc79e2564
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836484, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:44.678989 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:44.666772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:60889 (local address 127.7.194.254:34799)
0619 02:34:44.666907 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:44.666911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:44.667901 (+ 990us) server_negotiation.cc:407] Connection header received
0619 02:34:44.669320 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:44.669322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:44.669371 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:44.669437 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:44.670664 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.671204 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.672348 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.672568 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.672654 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:44.675338 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:44.675342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:44.675344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:44.675369 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:44.676738 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:44.677227 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:44.677230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:44.677231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:44.677310 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:44.677665 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:44.677668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:44.677670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:44.677911 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:44.678002 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:44.678611 (+ 609us) server_negotiation.cc:299] Negotiation successful
0619 02:34:44.678803 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:34:44.680050 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:44.667260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:34799 (local address 127.7.194.195:60889)
0619 02:34:44.667745 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:44.667808 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:44.669127 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:44.669633 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:44.669641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:44.670014 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:44.670531 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:44.670545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.671312 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.671315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:44.672245 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:44.672253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.672702 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.672704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:44.672864 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:44.673505 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:44.673521 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:44.675046 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:44.676861 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:44.676866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:44.676881 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:44.677134 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:44.677390 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:44.677392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:44.677393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:44.677528 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:44.678036 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:44.678042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:44.678337 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 02:34:44.678554 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 02:34:44.678937 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":144,"threads_started":1}
I20260619 02:34:44.681264 11483 heartbeater.cc:347] Connected to a master server at 127.7.194.254:34799
I20260619 02:34:44.681515 11483 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:44.682080 11483 heartbeater.cc:511] Master 127.7.194.254:34799 requested a full tablet report, sending...
I20260619 02:34:44.683120 11024 ts_manager.cc:194] Registered new tserver with Master: 6899cdb9809c43a4a04aa9b13eba3c19 (127.7.194.195:44243)
I20260619 02:34:44.683701 11024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:60889
I20260619 02:34:44.691948 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836484, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836484, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:44.717896 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:44.709416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.7.194.254:34799)
0619 02:34:44.709615 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:44.709620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:44.709814 (+ 194us) server_negotiation.cc:407] Connection header received
0619 02:34:44.709926 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:44.709928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:44.709974 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:44.710047 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:44.710755 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.711284 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.712100 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:44.712319 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:44.712402 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:44.714821 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:44.714824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:44.714826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:44.714854 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:44.716326 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:44.716685 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:44.716688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:44.716689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:44.716766 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:44.717019 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:44.717021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:44.717023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:44.717238 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:44.717340 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:44.717527 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 02:34:44.717679 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 02:34:44.718345 11024 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36354
I20260619 02:34:44.719173 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 11086
I20260619 02:34:44.724798 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 11220
I20260619 02:34:44.730284 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 11354
I20260619 02:34:44.735700 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 10992
2026-06-19T02:34:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2184 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[11495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[11495](info): set up 2 sockets
Jun 19 02:34:44 dist-test-slave-9tkz krb5kdc[11495](info): commencing operation
krb5kdc: starting...
W20260619 02:34:45.789836 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:45 dist-test-slave-9tkz krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:45.916347 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:42951
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:46.032040 11508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:46.032397 11508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:46.032482 11508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:46.036682 11508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:46.036801 11508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:46.036844 11508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:46.036901 11508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:46.036939 11508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:46.042737 11508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:42951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:46.043944 11508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:46.045061 11508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:46.053793 11516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:46.053855 11508 server_base.cc:1034] running on GCE node
W20260619 02:34:46.053793 11514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:46.053795 11513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:46.054715 11508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:46.055799 11508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:46.057001 11508 hybrid_clock.cc:648] HybridClock initialized: now 1781836486056981 us; error 31 us; skew 500 ppm
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:46.063880 11508 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:46.064881 11508 webserver.cc:460] Webserver started at http://127.7.194.254:42995/ using document root <none> and password file <none>
I20260619 02:34:46.065495 11508 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:46.065614 11508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:46.065881 11508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:46.068048 11508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "57dedb6a74ff405586680b6fe7864d06"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "144a3edd24db5087c43db70e7586a8dd"
server_key_iv: "5355b4a54719ba015f96950c0f4fa365"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.068638 11508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "57dedb6a74ff405586680b6fe7864d06"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "144a3edd24db5087c43db70e7586a8dd"
server_key_iv: "5355b4a54719ba015f96950c0f4fa365"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.072746 11508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 02:34:46.075440 11523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.076851 11508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:34:46.077010 11508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "57dedb6a74ff405586680b6fe7864d06"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "144a3edd24db5087c43db70e7586a8dd"
server_key_iv: "5355b4a54719ba015f96950c0f4fa365"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.077147 11508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:46.105753 11508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:46.109143 11508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:46.109396 11508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:46.117336 11508 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:42951
I20260619 02:34:46.117353 11575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:42951 every 8 connection(s)
I20260619 02:34:46.118413 11508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:46.121306 11576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:46.123198 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 11508
I20260619 02:34:46.123302 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:46.123581 7947 external_mini_cluster.cc:1428] Setting key 3e6014f70ef17aadee179d245fac82f7
I20260619 02:34:46.127846 11576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06: Bootstrap starting.
I20260619 02:34:46.130249 11576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:46.131058 11576 log.cc:824] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:46.133160 11576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06: No bootstrap required, opened a new log
I20260619 02:34:46.136729 11576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dedb6a74ff405586680b6fe7864d06" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42951 } }
I20260619 02:34:46.137014 11576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:46.137089 11576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57dedb6a74ff405586680b6fe7864d06, State: Initialized, Role: FOLLOWER
I20260619 02:34:46.137666 11576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dedb6a74ff405586680b6fe7864d06" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42951 } }
I20260619 02:34:46.137856 11576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:46.137933 11576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:46.138032 11576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:46.138641 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:46.125119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.7.194.254:42951)
0619 02:34:46.125842 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:46.125869 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:46.125923 (+ 54us) server_negotiation.cc:407] Connection header received
0619 02:34:46.126907 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:46.126929 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:46.127288 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:46.127648 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:46.128501 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.129518 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.130441 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.130951 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.131123 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:46.133741 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:46.133747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:46.133761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:46.133791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:46.135577 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.136056 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.136062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.136066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.136180 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.136471 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.136476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.136478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.136895 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:46.137116 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:46.137369 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 02:34:46.137713 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":141,"threads_started":1}
I20260619 02:34:46.139170 11576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dedb6a74ff405586680b6fe7864d06" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42951 } }
I20260619 02:34:46.139539 11576 leader_election.cc:302] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57dedb6a74ff405586680b6fe7864d06; no voters:
I20260619 02:34:46.139986 11576 leader_election.cc:288] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:46.140057 11581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:46.140283 11581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [term 1 LEADER]: Becoming Leader. State: Replica: 57dedb6a74ff405586680b6fe7864d06, State: Running, Role: LEADER
I20260619 02:34:46.140661 11581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dedb6a74ff405586680b6fe7864d06" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42951 } }
I20260619 02:34:46.141361 11576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:46.142628 11583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57dedb6a74ff405586680b6fe7864d06. Latest consensus state: current_term: 1 leader_uuid: "57dedb6a74ff405586680b6fe7864d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dedb6a74ff405586680b6fe7864d06" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42951 } } }
I20260619 02:34:46.142659 11582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57dedb6a74ff405586680b6fe7864d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dedb6a74ff405586680b6fe7864d06" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42951 } } }
I20260619 02:34:46.142868 11583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:46.142894 11582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:46.143260 11590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:46.146839 11590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:46.154034 11590 catalog_manager.cc:1347] Generated new cluster ID: 6b6adddade484d66902a5096c38b879a
I20260619 02:34:46.154125 11590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:46.161661 11590 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:46.162988 11590 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:46.173432 11590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57dedb6a74ff405586680b6fe7864d06: Generated new TSK 0
I20260619 02:34:46.174399 11590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:34:46.202697 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42951
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:46.320849 11602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:46.321151 11602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:46.321197 11602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:46.325445 11602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:46.325527 11602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:46.325608 11602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:46.331327 11602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:46.332506 11602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:46.333653 11602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:46.343734 11607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:46.343748 11608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:46.343869 11602 server_base.cc:1034] running on GCE node
W20260619 02:34:46.343758 11610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:46.344412 11602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:46.345089 11602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:46.346269 11602 hybrid_clock.cc:648] HybridClock initialized: now 1781836486346234 us; error 50 us; skew 500 ppm
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:46.349320 11602 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:34:46.350230 11602 webserver.cc:460] Webserver started at http://127.7.194.193:39009/ using document root <none> and password file <none>
I20260619 02:34:46.350783 11602 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:46.350843 11602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:46.351011 11602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:46.353063 11602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "3781a7d64f8b4abfa2af9afe9b84cf7d"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "749a91ca5c7d571b5d5f02ffb860ca88"
server_key_iv: "49ad495489bac13408b8960a91cca7c9"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.353660 11602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "3781a7d64f8b4abfa2af9afe9b84cf7d"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "749a91ca5c7d571b5d5f02ffb860ca88"
server_key_iv: "49ad495489bac13408b8960a91cca7c9"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.357476 11602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:34:46.360170 11617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.361505 11602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:34:46.361680 11602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "3781a7d64f8b4abfa2af9afe9b84cf7d"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "749a91ca5c7d571b5d5f02ffb860ca88"
server_key_iv: "49ad495489bac13408b8960a91cca7c9"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.361795 11602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:46.373193 11602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:46.376274 11602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:46.376498 11602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:46.377121 11602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:46.378079 11602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:46.378125 11602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.378196 11602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:46.378227 11602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.387324 11602 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:36927
I20260619 02:34:46.387341 11730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:36927 every 8 connection(s)
I20260619 02:34:46.388298 11602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:46.389815 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 11602
I20260619 02:34:46.389945 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:46.390287 7947 external_mini_cluster.cc:1428] Setting key 5eb0bbe076577d31777528d5924ae0a2
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:34:46.403918 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:46.390371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:55691 (local address 127.7.194.254:42951)
0619 02:34:46.390523 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:46.390527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:46.391566 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 02:34:46.393150 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:46.393154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:46.393209 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:46.393293 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:46.394604 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.395498 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.396791 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.397248 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.397389 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:46.400050 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:46.400053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:46.400055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:46.400081 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:46.401537 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.402121 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.402125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.402126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.402209 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.402565 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.402570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.402572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.402835 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:46.402947 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:46.403483 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 02:34:46.403651 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:34:46.404879 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:46.390864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:42951 (local address 127.7.194.193:55691)
0619 02:34:46.391391 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:46.391438 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:46.392899 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:46.393484 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:46.393492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:46.393923 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:46.394443 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:46.394457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.395635 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.395638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:46.396647 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:46.396657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.397405 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.397408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:46.397615 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:46.398269 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:46.398287 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:46.399867 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:46.401740 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:46.401746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:46.401763 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:46.402020 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:46.402302 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:46.402306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:46.402308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:46.402484 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:46.402971 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:46.402976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:46.403231 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 02:34:46.403407 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 02:34:46.403723 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":155,"threads_started":1}
I20260619 02:34:46.406260 11731 heartbeater.cc:347] Connected to a master server at 127.7.194.254:42951
I20260619 02:34:46.406589 11731 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:46.407260 11731 heartbeater.cc:511] Master 127.7.194.254:42951 requested a full tablet report, sending...
I20260619 02:34:46.409030 11540 ts_manager.cc:194] Registered new tserver with Master: 3781a7d64f8b4abfa2af9afe9b84cf7d (127.7.194.193:36927)
I20260619 02:34:46.410264 11540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:55691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:34:46.418913 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42951
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:46.535319 11736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:46.535655 11736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:46.535749 11736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:46.540308 11736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:46.540411 11736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:46.540503 11736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:46.546664 11736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:46.547861 11736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:46.549033 11736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:46.559404 11744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:46.559423 11736 server_base.cc:1034] running on GCE node
W20260619 02:34:46.559487 11741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:46.559404 11742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:46.560191 11736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:46.560844 11736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:46.562032 11736 hybrid_clock.cc:648] HybridClock initialized: now 1781836486562009 us; error 35 us; skew 500 ppm
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:46.564946 11736 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:34:46.565974 11736 webserver.cc:460] Webserver started at http://127.7.194.194:44555/ using document root <none> and password file <none>
I20260619 02:34:46.566553 11736 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:46.566655 11736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:46.566895 11736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:46.569077 11736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "7b29b6ea58904b198ca3cc53a792b2d2"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "feff8e743bd34a35b33d50ad0c4651e0"
server_key_iv: "1c0c27cc19c4f7b94725219f754bbc31"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.569764 11736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "7b29b6ea58904b198ca3cc53a792b2d2"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "feff8e743bd34a35b33d50ad0c4651e0"
server_key_iv: "1c0c27cc19c4f7b94725219f754bbc31"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.573776 11736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:46.576664 11751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.578069 11736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:46.578224 11736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "7b29b6ea58904b198ca3cc53a792b2d2"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "feff8e743bd34a35b33d50ad0c4651e0"
server_key_iv: "1c0c27cc19c4f7b94725219f754bbc31"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.578349 11736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:46.599434 11736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:46.603272 11736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:46.603508 11736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:46.604185 11736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:46.605121 11736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:46.605199 11736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.605279 11736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:46.605335 11736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.615492 11736 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:44649
I20260619 02:34:46.615510 11864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:44649 every 8 connection(s)
I20260619 02:34:46.616492 11736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:46.619730 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 11736
I20260619 02:34:46.619871 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:46.620190 7947 external_mini_cluster.cc:1428] Setting key d4d5a45e11f9601f99177a87266c7bca
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:34:46.632190 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:46.618535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:59905 (local address 127.7.194.254:42951)
0619 02:34:46.618706 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:46.618710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:46.619716 (+ 1006us) server_negotiation.cc:407] Connection header received
0619 02:34:46.621404 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:46.621407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:46.621466 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:46.621577 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:46.622890 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.623470 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.624759 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.625010 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.625136 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:46.628012 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:46.628016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:46.628018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:46.628045 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:46.629801 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.630406 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.630409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.630411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.630496 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.630903 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.630909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.630916 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.631109 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:46.631229 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:46.631864 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 02:34:46.632005 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:34:46.633388 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:46.618984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:42951 (local address 127.7.194.194:59905)
0619 02:34:46.619503 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:46.619550 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:46.621166 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:46.621774 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:46.621783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:46.622190 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:46.622724 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:46.622739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.623615 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.623620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:46.624616 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:46.624626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.625187 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.625192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:46.625389 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:46.626210 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:46.626229 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:46.627815 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:46.629928 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:46.629935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:46.629956 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:46.630258 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:46.630606 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:46.630610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:46.630612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:46.630769 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:46.631231 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:46.631240 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:46.631521 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 02:34:46.631778 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 02:34:46.632194 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
I20260619 02:34:46.634986 11865 heartbeater.cc:347] Connected to a master server at 127.7.194.254:42951
I20260619 02:34:46.635291 11865 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:46.635844 11865 heartbeater.cc:511] Master 127.7.194.254:42951 requested a full tablet report, sending...
I20260619 02:34:46.637095 11540 ts_manager.cc:194] Registered new tserver with Master: 7b29b6ea58904b198ca3cc53a792b2d2 (127.7.194.194:44649)
I20260619 02:34:46.638092 11540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:59905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:34:46.650736 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42951
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:46.763592 11870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:46.763919 11870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:46.763998 11870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:46.768275 11870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:46.768373 11870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:46.768469 11870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:46.773968 11870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:46.775194 11870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:46.776294 11870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:46.786378 11878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:46.786370 11875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:46.786454 11870 server_base.cc:1034] running on GCE node
W20260619 02:34:46.786370 11876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:46.787110 11870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:46.787768 11870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:46.788966 11870 hybrid_clock.cc:648] HybridClock initialized: now 1781836486788945 us; error 31 us; skew 500 ppm
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:46.792121 11870 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:34:46.793005 11870 webserver.cc:460] Webserver started at http://127.7.194.195:33227/ using document root <none> and password file <none>
I20260619 02:34:46.793649 11870 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:46.793761 11870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:46.794001 11870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:46.796092 11870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "db05ade966ca46d7a2668313f2b3db78"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "b98095a97e14d058782a796b801aeb1d"
server_key_iv: "a990a138207d141db0235933c0cb59fa"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.796693 11870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "db05ade966ca46d7a2668313f2b3db78"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "b98095a97e14d058782a796b801aeb1d"
server_key_iv: "a990a138207d141db0235933c0cb59fa"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.800541 11870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 02:34:46.803220 11885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.804489 11870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:46.804628 11870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "db05ade966ca46d7a2668313f2b3db78"
format_stamp: "Formatted at 2026-06-19 02:34:46 on dist-test-slave-9tkz"
server_key: "b98095a97e14d058782a796b801aeb1d"
server_key_iv: "a990a138207d141db0235933c0cb59fa"
server_key_version: "encryptionkey@0"
I20260619 02:34:46.804756 11870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:46.826932 11870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:46.830365 11870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:46.830583 11870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:46.831211 11870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:46.832067 11870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:46.832137 11870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.832207 11870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:46.832259 11870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:46.841131 11870 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:45727
I20260619 02:34:46.841142 11998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:45727 every 8 connection(s)
I20260619 02:34:46.842149 11870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:46.847425 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 11870
I20260619 02:34:46.847585 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:46.847922 7947 external_mini_cluster.cc:1428] Setting key 93aabf83543efa7252005341aa30c137
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:46.856878 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:46.844159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:41679 (local address 127.7.194.254:42951)
0619 02:34:46.844309 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:46.844313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:46.845222 (+ 909us) server_negotiation.cc:407] Connection header received
0619 02:34:46.846687 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:46.846691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:46.846748 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:46.846868 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:46.848249 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.848941 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.850262 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.850503 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.850617 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:46.853258 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:46.853262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:46.853264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:46.853287 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:46.854798 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.855329 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.855337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.855339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.855413 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.855767 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.855770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.855772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.855958 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:46.856066 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:46.856575 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 02:34:46.856691 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:34:46.857908 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:46.844596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:42951 (local address 127.7.194.195:41679)
0619 02:34:46.845064 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:46.845105 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:46.846465 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:46.847045 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:46.847053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:46.847445 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:46.848084 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:46.848108 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.849067 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.849070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:46.850142 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:46.850152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.850652 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.850655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:46.850868 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:46.851559 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:46.851575 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:46.853069 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:46.854924 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:46.854931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:46.854947 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:46.855216 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:46.855503 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:46.855508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:46.855510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:46.855658 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:46.856062 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:46.856070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:46.856332 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 02:34:46.856536 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 02:34:46.856924 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260619 02:34:46.859074 11999 heartbeater.cc:347] Connected to a master server at 127.7.194.254:42951
I20260619 02:34:46.859329 11999 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:46.859833 11999 heartbeater.cc:511] Master 127.7.194.254:42951 requested a full tablet report, sending...
I20260619 02:34:46.860985 11540 ts_manager.cc:194] Registered new tserver with Master: db05ade966ca46d7a2668313f2b3db78 (127.7.194.195:45727)
I20260619 02:34:46.861799 11540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:41679
I20260619 02:34:46.861867 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:46.888921 12006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:46.879246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.7.194.193:36927)
0619 02:34:46.879579 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:46.879598 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:46.879628 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:34:46.879701 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:46.879708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:46.879914 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:46.880076 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:46.880818 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.881711 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.882594 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.882993 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.883115 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:46.885644 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:46.885653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:46.885667 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:46.885693 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:46.887213 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.887614 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.887620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.887621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.887699 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.887933 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.887935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.887936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.888180 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:46.888347 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:46.888534 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 02:34:46.888696 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":121,"threads_started":1}
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:46.900424 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:46.891888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.7.194.254:42951)
0619 02:34:46.892025 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:46.892031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:46.892103 (+ 72us) server_negotiation.cc:407] Connection header received
0619 02:34:46.892257 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:46.892261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:46.892310 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:46.892401 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:46.893002 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.893537 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.894380 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:46.894621 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:46.894731 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:46.897146 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:46.897149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:46.897151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:46.897175 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:46.898669 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.899103 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.899107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.899109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.899196 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:46.899483 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:46.899487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:46.899489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:46.899728 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:46.899874 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:46.900111 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 02:34:46.900237 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 02:34:46.900862 11540 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44194
I20260619 02:34:46.901631 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 11602
I20260619 02:34:46.907517 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 11736
I20260619 02:34:46.913020 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 11870
I20260619 02:34:46.918510 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 11508
2026-06-19T02:34:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2182 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[12011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[12011](info): set up 2 sockets
Jun 19 02:34:46 dist-test-slave-9tkz krb5kdc[12011](info): commencing operation
krb5kdc: starting...
W20260619 02:34:47.970078 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:48 dist-test-slave-9tkz krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:48.096295 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:36471
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:36471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:48.212548 12024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:48.212849 12024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:48.212894 12024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:48.217028 12024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:48.217103 12024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:48.217121 12024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:48.217139 12024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:48.217203 12024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:48.222856 12024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:36471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:36471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:48.224025 12024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:48.225131 12024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:48.233820 12032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:48.233835 12029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:48.233865 12030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:48.234076 12024 server_base.cc:1034] running on GCE node
I20260619 02:34:48.234742 12024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:48.235888 12024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:48.237112 12024 hybrid_clock.cc:648] HybridClock initialized: now 1781836488237078 us; error 40 us; skew 500 ppm
Jun 19 02:34:48 dist-test-slave-9tkz krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836488, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:48.240078 12024 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:48.240870 12024 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:48.240972 12024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:48.241235 12024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:48.243449 12024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "4bd3950cc4f5464db017e7d6690f986c"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "398dad56817b03cee2aea27a2776c451"
server_key_iv: "927db5aa382cf538d7738e750d1915a8"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.244043 12024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "4bd3950cc4f5464db017e7d6690f986c"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "398dad56817b03cee2aea27a2776c451"
server_key_iv: "927db5aa382cf538d7738e750d1915a8"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.247952 12024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 02:34:48.250553 12038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:48.252051 12024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:48.252192 12024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "4bd3950cc4f5464db017e7d6690f986c"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "398dad56817b03cee2aea27a2776c451"
server_key_iv: "927db5aa382cf538d7738e750d1915a8"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.252353 12024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:48.271781 12024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:48.275400 12024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:48.275622 12024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:48.282825 12024 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:36471
I20260619 02:34:48.283141 12090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:36471 every 8 connection(s)
I20260619 02:34:48.283531 12024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:48.286437 12091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:48.292685 12091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c: Bootstrap starting.
I20260619 02:34:48.293143 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 12024
I20260619 02:34:48.293272 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:48.293593 7947 external_mini_cluster.cc:1428] Setting key 13a7877cab5129e4c88488500d5cee7b
I20260619 02:34:48.295593 12091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:48.296376 12091 log.cc:824] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:48.298712 12091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c: No bootstrap required, opened a new log
Jun 19 02:34:48 dist-test-slave-9tkz krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:48.303345 12091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3950cc4f5464db017e7d6690f986c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 36471 } }
I20260619 02:34:48.303629 12091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:48.303695 12091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd3950cc4f5464db017e7d6690f986c, State: Initialized, Role: FOLLOWER
I20260619 02:34:48.304244 12091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3950cc4f5464db017e7d6690f986c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 36471 } }
I20260619 02:34:48.304419 12091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:48.304492 12091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:48.304644 12091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:48.305943 12091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3950cc4f5464db017e7d6690f986c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 36471 } }
I20260619 02:34:48.306347 12091 leader_election.cc:302] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd3950cc4f5464db017e7d6690f986c; no voters:
I20260619 02:34:48.306747 12091 leader_election.cc:288] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:48.306818 12096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:48.307013 12096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [term 1 LEADER]: Becoming Leader. State: Replica: 4bd3950cc4f5464db017e7d6690f986c, State: Running, Role: LEADER
I20260619 02:34:48.307392 12096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3950cc4f5464db017e7d6690f986c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 36471 } }
I20260619 02:34:48.308120 12091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:48.309122 12098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bd3950cc4f5464db017e7d6690f986c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3950cc4f5464db017e7d6690f986c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 36471 } } }
I20260619 02:34:48.309242 12098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:48.309684 12105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:48.309661 12097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bd3950cc4f5464db017e7d6690f986c. Latest consensus state: current_term: 1 leader_uuid: "4bd3950cc4f5464db017e7d6690f986c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3950cc4f5464db017e7d6690f986c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 36471 } } }
I20260619 02:34:48.309751 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:48.295016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.7.194.254:36471)
0619 02:34:48.295412 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:48.295430 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:48.295477 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:34:48.296171 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:48.296189 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:48.296615 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:48.296987 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:48.297968 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.298996 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.300177 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.300645 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.300768 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:48.303602 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:48.303608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:48.303625 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:48.303660 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:48.305818 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:48.306290 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:48.306296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:48.306301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:48.306449 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:48.306755 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:48.306758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:48.306760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:48.307343 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:48.308105 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:48.308271 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 02:34:48.308583 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":138,"threads_started":1}
I20260619 02:34:48.309818 12097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:48.313140 12105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:48.318956 12105 catalog_manager.cc:1347] Generated new cluster ID: 6d002de97cba4006aaef35aec7e2004c
I20260619 02:34:48.319046 12105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:48.341244 12105 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:48.342368 12105 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:48.346685 12105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c: Generated new TSK 0
I20260619 02:34:48.347599 12105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:34:48.382417 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:36471
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:48.499109 12117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:48.499418 12117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:48.499464 12117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:48.503486 12117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:48.503556 12117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:48.503619 12117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:48.509234 12117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:36471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:48.510465 12117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:48.511520 12117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:48.521183 12125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:48.521234 12122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:48.521188 12123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:48.521504 12117 server_base.cc:1034] running on GCE node
I20260619 02:34:48.521998 12117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:48.522639 12117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:48.523810 12117 hybrid_clock.cc:648] HybridClock initialized: now 1781836488523793 us; error 31 us; skew 500 ppm
Jun 19 02:34:48 dist-test-slave-9tkz krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836488, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:48.526804 12117 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:34:48.527531 12117 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:48.527627 12117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:48.527868 12117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:48.530006 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "576e7ac54cf546a78b461c9526008c15"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "90ac6dc4f82992ae771b9d56ace3f821"
server_key_iv: "517c8638a1bea630da12d3f0ef82329c"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.530591 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "576e7ac54cf546a78b461c9526008c15"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "90ac6dc4f82992ae771b9d56ace3f821"
server_key_iv: "517c8638a1bea630da12d3f0ef82329c"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.534389 12117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:48.536998 12131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:48.538400 12117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:48.538553 12117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "576e7ac54cf546a78b461c9526008c15"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "90ac6dc4f82992ae771b9d56ace3f821"
server_key_iv: "517c8638a1bea630da12d3f0ef82329c"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.538683 12117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:48.566057 12117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:48.570166 12117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:48.570390 12117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:48.571027 12117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:48.571919 12117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:48.571988 12117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:48.572044 12117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:48.572093 12117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:48.581152 12117 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:42547
I20260619 02:34:48.581197 12244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:42547 every 8 connection(s)
I20260619 02:34:48.581878 12117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:48.589505 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 12117
I20260619 02:34:48.589671 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:48.589954 7947 external_mini_cluster.cc:1428] Setting key ba8647eed203b8845d31b77c86c9d20b
Jun 19 02:34:48 dist-test-slave-9tkz krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836488, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:48.599030 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:48.584192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:40031 (local address 127.7.194.254:36471)
0619 02:34:48.584370 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:48.584377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:48.585598 (+ 1221us) server_negotiation.cc:407] Connection header received
0619 02:34:48.587514 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:48.587520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:48.587599 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:48.587713 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:48.589454 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.590239 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.591719 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.591977 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.592110 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:48.594877 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:48.594881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:48.594883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:48.594910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:48.596650 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:48.597224 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:48.597230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:48.597232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:48.597329 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:48.597790 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:48.597795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:48.597797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:48.598005 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:48.598121 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:48.598686 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 02:34:48.598832 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:34:48.599912 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:48.584713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:36471 (local address 127.7.194.193:40031)
0619 02:34:48.585356 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:48.585432 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:48.587265 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:48.588048 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:48.588057 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:48.588620 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:48.589280 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:48.589297 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.590395 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.590399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:48.591571 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:48.591582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.592139 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.592144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:48.592339 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:48.593026 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:48.593051 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:48.594680 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:48.596783 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:48.596790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:48.596807 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:48.597106 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:48.597444 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:48.597448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:48.597450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:48.597682 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:48.598137 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:48.598146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:48.598415 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 02:34:48.598618 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 02:34:48.598977 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":174,"threads_started":1}
I20260619 02:34:48.601346 12245 heartbeater.cc:347] Connected to a master server at 127.7.194.254:36471
I20260619 02:34:48.601708 12245 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:34:48.602267 12245 heartbeater.cc:511] Master 127.7.194.254:36471 requested a full tablet report, sending...
I20260619 02:34:48.603940 12055 ts_manager.cc:194] Registered new tserver with Master: 576e7ac54cf546a78b461c9526008c15 (127.7.194.193:42547)
I20260619 02:34:48.605167 12055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:40031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:34:48.617240 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:36471
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:48.728976 12250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:48.729278 12250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:48.729326 12250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:48.733467 12250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:48.733531 12250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:48.733608 12250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:48.738942 12250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:36471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:48.740124 12250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:48.741276 12250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:48.750864 12256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:48.750885 12258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:48.750922 12250 server_base.cc:1034] running on GCE node
W20260619 02:34:48.750886 12255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:48.751504 12250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:48.752174 12250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:48.753398 12250 hybrid_clock.cc:648] HybridClock initialized: now 1781836488753385 us; error 28 us; skew 500 ppm
Jun 19 02:34:48 dist-test-slave-9tkz krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836488, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:48.756297 12250 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:34:48.757135 12250 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:48.757225 12250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:48.757436 12250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:48.759581 12250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "ba04ea6b37bd4193841d345f512cdf21"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "bf217718550fb02fdd2d3e40201fd8ea"
server_key_iv: "cd9a50996dbe0faf6eb4c3f39d229e71"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.760160 12250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "ba04ea6b37bd4193841d345f512cdf21"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "bf217718550fb02fdd2d3e40201fd8ea"
server_key_iv: "cd9a50996dbe0faf6eb4c3f39d229e71"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.764017 12250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:48.766515 12264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:48.767843 12250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:34:48.767978 12250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "ba04ea6b37bd4193841d345f512cdf21"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "bf217718550fb02fdd2d3e40201fd8ea"
server_key_iv: "cd9a50996dbe0faf6eb4c3f39d229e71"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.768098 12250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:48.789637 12250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:48.793891 12250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:48.794111 12250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:48.794848 12250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:48.795730 12250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:48.795778 12250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:48.795816 12250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:48.795831 12250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:48.805384 12250 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:46379
I20260619 02:34:48.805400 12377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:46379 every 8 connection(s)
I20260619 02:34:48.806044 12250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:48.813823 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 12250
I20260619 02:34:48.813944 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:48.814240 7947 external_mini_cluster.cc:1428] Setting key 950b5d327f259a05f707146a0a35f2c0
Jun 19 02:34:48 dist-test-slave-9tkz krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836488, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:48.820654 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:48.807826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:36353 (local address 127.7.194.254:36471)
0619 02:34:48.807966 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:48.807970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:48.808865 (+ 895us) server_negotiation.cc:407] Connection header received
0619 02:34:48.810356 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:48.810360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:48.810421 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:48.810503 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:48.811710 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.812294 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.813491 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.813863 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.813995 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:48.816947 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:48.816951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:48.816954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:48.816983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:48.818542 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:48.819086 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:48.819091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:48.819092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:48.819184 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:48.819539 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:48.819542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:48.819544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:48.819733 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:48.819836 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:48.820363 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 02:34:48.820486 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:34:48.821534 12380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:48.808183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:36471 (local address 127.7.194.194:36353)
0619 02:34:48.808712 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:48.808757 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:48.810121 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:48.810680 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:48.810688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:48.811055 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:48.811553 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:48.811568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.812437 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.812441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:48.813353 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:48.813362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:48.814038 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:48.814043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:48.814228 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:48.815109 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:48.815127 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:48.816761 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:48.818669 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:48.818675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:48.818693 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:48.818976 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:48.819277 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:48.819281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:48.819283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:48.819433 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:48.819835 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:48.819844 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:48.820116 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 02:34:48.820313 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 02:34:48.820679 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":120,"threads_started":1}
I20260619 02:34:48.822819 12378 heartbeater.cc:347] Connected to a master server at 127.7.194.254:36471
I20260619 02:34:48.823124 12378 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:34:48.823725 12378 heartbeater.cc:511] Master 127.7.194.254:36471 requested a full tablet report, sending...
I20260619 02:34:48.824952 12055 ts_manager.cc:194] Registered new tserver with Master: ba04ea6b37bd4193841d345f512cdf21 (127.7.194.194:46379)
I20260619 02:34:48.825739 12055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:36353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:34:48.842360 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:36471
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:48.957301 12383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:48.957641 12383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:48.957701 12383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:48.961934 12383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:48.962002 12383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:48.962105 12383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:48.967713 12383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:36471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:48.968914 12383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:48.970047 12383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:48.980268 12389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:48.980268 12388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:48.980288 12391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:48.980710 12383 server_base.cc:1034] running on GCE node
I20260619 02:34:48.981189 12383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:48.982009 12383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:48.983196 12383 hybrid_clock.cc:648] HybridClock initialized: now 1781836488983180 us; error 29 us; skew 500 ppm
Jun 19 02:34:48 dist-test-slave-9tkz krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836488, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:48.986402 12383 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:34:48.987174 12383 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:48.987243 12383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:48.987427 12383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:48.989714 12383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "d842bbb29fae41959d4d1b0f21d1253f"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "b18a01eef7bea0732ed6d98a3205e342"
server_key_iv: "b5627b1285e9ecf35d4579c604bb73d9"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.990295 12383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "d842bbb29fae41959d4d1b0f21d1253f"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "b18a01eef7bea0732ed6d98a3205e342"
server_key_iv: "b5627b1285e9ecf35d4579c604bb73d9"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.994349 12383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:34:48.996897 12397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:48.998261 12383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:34:48.998406 12383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "d842bbb29fae41959d4d1b0f21d1253f"
format_stamp: "Formatted at 2026-06-19 02:34:48 on dist-test-slave-9tkz"
server_key: "b18a01eef7bea0732ed6d98a3205e342"
server_key_iv: "b5627b1285e9ecf35d4579c604bb73d9"
server_key_version: "encryptionkey@0"
I20260619 02:34:48.998528 12383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:49.013496 12383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:49.017048 12383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:49.017271 12383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:49.017932 12383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:49.018802 12383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:49.018880 12383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:49.018968 12383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:49.019017 12383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:49.028508 12383 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:42031
I20260619 02:34:49.028525 12510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:42031 every 8 connection(s)
I20260619 02:34:49.029151 12383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:49.029364 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 12383
I20260619 02:34:49.029500 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:49.029877 7947 external_mini_cluster.cc:1428] Setting key 9ba02bc4dd948a5904fcf3a0182fc968
Jun 19 02:34:49 dist-test-slave-9tkz krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836488, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:49.042834 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.031165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:49663 (local address 127.7.194.254:36471)
0619 02:34:49.031299 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.031304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.032295 (+ 991us) server_negotiation.cc:407] Connection header received
0619 02:34:49.033612 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.033615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.033670 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.033741 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:49.034872 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.035394 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.036554 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.036781 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.036894 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.039338 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:49.039341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:49.039343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:49.039368 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:49.040701 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:49.041201 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:49.041205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:49.041207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:49.041286 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:49.041652 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:49.041655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:49.041656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:49.041877 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:49.041961 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.042520 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.042666 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:34:49.043865 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.031676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:36471 (local address 127.7.194.195:49663)
0619 02:34:49.032145 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:49.032197 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:49.033388 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:49.033883 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:49.033890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:49.034251 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:49.034750 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.034764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.035496 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.035498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.036420 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.036429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.036934 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.036938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.037127 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.037745 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:49.037760 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:49.039139 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:49.040828 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:49.040833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:49.040850 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:49.041097 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:49.041381 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:49.041384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:49.041385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:49.041521 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:49.041971 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:49.041977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:49.042223 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 02:34:49.042476 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 02:34:49.042840 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":132,"threads_started":1}
I20260619 02:34:49.044926 12511 heartbeater.cc:347] Connected to a master server at 127.7.194.254:36471
I20260619 02:34:49.045184 12511 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:49.045722 12511 heartbeater.cc:511] Master 127.7.194.254:36471 requested a full tablet report, sending...
I20260619 02:34:49.046675 12055 ts_manager.cc:194] Registered new tserver with Master: d842bbb29fae41959d4d1b0f21d1253f (127.7.194.195:42031)
I20260619 02:34:49.047273 12055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:49663
I20260619 02:34:49.055873 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:34:49.065274 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.057916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.7.194.254:36471)
0619 02:34:49.058035 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.058039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.058231 (+ 192us) server_negotiation.cc:407] Connection header received
0619 02:34:49.058329 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.058331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.058371 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.058428 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:49.059118 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.059630 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.060425 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.060625 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.060711 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.062152 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:49.062157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:49.062161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:49.062194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:49.063784 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:49.064162 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:49.064164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:49.064166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:49.064235 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:49.064440 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:49.064441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:49.064442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:49.064711 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:49.064818 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.065052 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.065147 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:34:49.068275 12055 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:34:49.071640 12055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:49.086879 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.080426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:42031 (local address 127.0.0.1:44790)
0619 02:34:49.082107 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:49.082115 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:49.082209 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:49.082683 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:49.082685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:49.082700 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:49.082774 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.082779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.084132 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.084135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.085197 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.085207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.086294 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.086298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.086403 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.086444 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:34:49.086567 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 02:34:49.086712 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"thread_start_us":85,"threads_started":1}
I20260619 02:34:49.087062 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.080483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:46379 (local address 127.0.0.1:39444)
0619 02:34:49.081634 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:49.081690 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:49.081814 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:49.082388 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:49.082393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:49.082415 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:49.082547 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.082553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.084059 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.084064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.085684 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.085695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.086706 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.086709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.086788 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.086808 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 02:34:49.086853 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 02:34:49.086931 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":190,"thread_start_us":128,"threads_started":1}
I20260619 02:34:49.087049 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.080544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.7.194.195:42031)
0619 02:34:49.081258 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.081271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.082197 (+ 926us) server_negotiation.cc:407] Connection header received
0619 02:34:49.082348 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.082354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.082533 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.082672 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:49.083026 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.083989 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.085319 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.086151 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.086267 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.086414 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.086610 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.086730 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":94,"threads_started":1}
I20260619 02:34:49.088090 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.081238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.7.194.194:46379)
0619 02:34:49.081653 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.081664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.081790 (+ 126us) server_negotiation.cc:407] Connection header received
0619 02:34:49.081953 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.081957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.082188 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.082390 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:49.082668 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.083909 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.085862 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.086586 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.087638 (+ 1052us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.087783 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.087851 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.087943 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":136,"threads_started":1}
I20260619 02:34:49.088289 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.080741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:42547 (local address 127.0.0.1:49708)
0619 02:34:49.081969 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:49.081977 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:49.082035 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:49.082555 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:49.082558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:49.082575 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:49.082703 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.082712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.084077 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.084080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.085690 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.085696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.086979 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.086983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.087976 (+ 993us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.087995 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:34:49.088086 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 02:34:49.088156 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"mutex_wait_us":135,"thread_start_us":78,"threads_started":1}
I20260619 02:34:49.088369 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.081240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.7.194.193:42547)
0619 02:34:49.081653 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.081662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.082057 (+ 395us) server_negotiation.cc:407] Connection header received
0619 02:34:49.082129 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.082133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.082293 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.082413 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:49.082830 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.083946 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.085843 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.086654 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.086764 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.086897 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.088120 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.088219 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":110,"threads_started":1}
I20260619 02:34:49.090399 12445 tablet_service.cc:1467] Processing CreateTablet for tablet 686eec692a314f7cae8e2fa63050ec97 (DEFAULT_TABLE table=test-table [id=174cfb24848b45078273a2935b747cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:49.090751 12312 tablet_service.cc:1467] Processing CreateTablet for tablet 686eec692a314f7cae8e2fa63050ec97 (DEFAULT_TABLE table=test-table [id=174cfb24848b45078273a2935b747cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:49.091630 12445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686eec692a314f7cae8e2fa63050ec97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:49.091305 12179 tablet_service.cc:1467] Processing CreateTablet for tablet 686eec692a314f7cae8e2fa63050ec97 (DEFAULT_TABLE table=test-table [id=174cfb24848b45078273a2935b747cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:49.091708 12312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686eec692a314f7cae8e2fa63050ec97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:49.092203 12179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686eec692a314f7cae8e2fa63050ec97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:49.097944 12528 tablet_bootstrap.cc:492] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: Bootstrap starting.
I20260619 02:34:49.099457 12530 tablet_bootstrap.cc:492] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: Bootstrap starting.
I20260619 02:34:49.100972 12528 tablet_bootstrap.cc:654] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:49.101341 12530 tablet_bootstrap.cc:654] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:49.101961 12529 tablet_bootstrap.cc:492] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15: Bootstrap starting.
I20260619 02:34:49.102237 12528 log.cc:824] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:49.102687 12530 log.cc:824] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:49.104122 12529 tablet_bootstrap.cc:654] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:49.104598 12528 tablet_bootstrap.cc:492] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: No bootstrap required, opened a new log
I20260619 02:34:49.104705 12530 tablet_bootstrap.cc:492] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: No bootstrap required, opened a new log
I20260619 02:34:49.104863 12530 ts_tablet_manager.cc:1397] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:34:49.104982 12529 log.cc:824] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:49.105135 12528 ts_tablet_manager.cc:1397] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260619 02:34:49.106624 12529 tablet_bootstrap.cc:492] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15: No bootstrap required, opened a new log
I20260619 02:34:49.106822 12529 ts_tablet_manager.cc:1397] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:34:49.108945 12528 raft_consensus.cc:348] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } }
I20260619 02:34:49.108945 12530 raft_consensus.cc:348] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } }
I20260619 02:34:49.109266 12530 raft_consensus.cc:374] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:49.109266 12528 raft_consensus.cc:374] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:49.109330 12528 raft_consensus.cc:729] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba04ea6b37bd4193841d345f512cdf21, State: Initialized, Role: FOLLOWER
I20260619 02:34:49.109361 12530 raft_consensus.cc:729] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d842bbb29fae41959d4d1b0f21d1253f, State: Initialized, Role: FOLLOWER
I20260619 02:34:49.109869 12530 consensus_queue.cc:260] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } }
I20260619 02:34:49.109869 12528 consensus_queue.cc:260] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } }
I20260619 02:34:49.110462 12529 raft_consensus.cc:348] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } }
I20260619 02:34:49.110705 12529 raft_consensus.cc:374] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:49.110788 12529 raft_consensus.cc:729] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576e7ac54cf546a78b461c9526008c15, State: Initialized, Role: FOLLOWER
I20260619 02:34:49.110975 12530 ts_tablet_manager.cc:1428] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:34:49.110975 12528 ts_tablet_manager.cc:1428] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:34:49.111263 12529 consensus_queue.cc:260] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } }
I20260619 02:34:49.111349 12511 heartbeater.cc:503] Master 127.7.194.254:36471 was elected leader, sending a full tablet report...
I20260619 02:34:49.111384 12378 heartbeater.cc:503] Master 127.7.194.254:36471 was elected leader, sending a full tablet report...
I20260619 02:34:49.112327 12529 ts_tablet_manager.cc:1428] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:34:49.112371 12245 heartbeater.cc:503] Master 127.7.194.254:36471 was elected leader, sending a full tablet report...
I20260619 02:34:49.200297 12534 raft_consensus.cc:482] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:49.200551 12534 raft_consensus.cc:504] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } }
I20260619 02:34:49.201954 12534 leader_election.cc:288] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba04ea6b37bd4193841d345f512cdf21 (127.7.194.194:46379), 576e7ac54cf546a78b461c9526008c15 (127.7.194.193:42547)
I20260619 02:34:49.206465 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.202442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:50293 (local address 127.7.194.193:42547)
0619 02:34:49.202564 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.202571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.202829 (+ 258us) server_negotiation.cc:407] Connection header received
0619 02:34:49.202914 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.202917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.202963 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.203042 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:49.203370 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.204116 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.205227 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.205870 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.205974 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.206016 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.206251 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.206332 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:34:49.206465 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.202328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:55513 (local address 127.7.194.194:46379)
0619 02:34:49.202472 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.202477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.202610 (+ 133us) server_negotiation.cc:407] Connection header received
0619 02:34:49.202722 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.202727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.202813 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.202893 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:49.203186 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.203782 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.205055 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.205756 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.205877 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.205952 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.206251 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.206332 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:34:49.206468 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.202186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:46379 (local address 127.7.194.195:55513)
0619 02:34:49.202441 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:49.202460 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:49.202591 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:49.202911 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:49.202913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:49.202933 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:49.203070 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.203078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.203894 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.203897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.204929 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.204939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.205921 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.205928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.206045 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.206118 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 02:34:49.206194 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:34:49.206317 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":93}
I20260619 02:34:49.206481 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.202311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:42547 (local address 127.7.194.195:50293)
0619 02:34:49.202704 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:49.202718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:49.202823 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:49.203077 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:49.203080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:49.203094 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:49.203235 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.203242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.204239 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.204241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.205095 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:49.205104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.205968 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.205971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:49.206050 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.206104 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:34:49.206203 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:34:49.206306 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":44,"thread_start_us":149,"threads_started":1}
I20260619 02:34:49.207151 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686eec692a314f7cae8e2fa63050ec97" candidate_uuid: "d842bbb29fae41959d4d1b0f21d1253f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576e7ac54cf546a78b461c9526008c15" is_pre_election: true
I20260619 02:34:49.207440 12332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686eec692a314f7cae8e2fa63050ec97" candidate_uuid: "d842bbb29fae41959d4d1b0f21d1253f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba04ea6b37bd4193841d345f512cdf21" is_pre_election: true
I20260619 02:34:49.207516 12199 raft_consensus.cc:2454] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d842bbb29fae41959d4d1b0f21d1253f in term 0.
I20260619 02:34:49.207739 12332 raft_consensus.cc:2454] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d842bbb29fae41959d4d1b0f21d1253f in term 0.
I20260619 02:34:49.208050 12401 leader_election.cc:302] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576e7ac54cf546a78b461c9526008c15, d842bbb29fae41959d4d1b0f21d1253f; no voters:
I20260619 02:34:49.208364 12534 raft_consensus.cc:2781] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:49.208455 12534 raft_consensus.cc:482] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:49.208511 12534 raft_consensus.cc:3037] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:49.209723 12534 raft_consensus.cc:504] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } }
I20260619 02:34:49.210124 12534 leader_election.cc:288] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [CANDIDATE]: Term 1 election: Requested vote from peers ba04ea6b37bd4193841d345f512cdf21 (127.7.194.194:46379), 576e7ac54cf546a78b461c9526008c15 (127.7.194.193:42547)
I20260619 02:34:49.210474 12332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686eec692a314f7cae8e2fa63050ec97" candidate_uuid: "d842bbb29fae41959d4d1b0f21d1253f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba04ea6b37bd4193841d345f512cdf21"
I20260619 02:34:49.210559 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686eec692a314f7cae8e2fa63050ec97" candidate_uuid: "d842bbb29fae41959d4d1b0f21d1253f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576e7ac54cf546a78b461c9526008c15"
I20260619 02:34:49.210624 12332 raft_consensus.cc:3037] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:49.210708 12199 raft_consensus.cc:3037] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:49.211786 12332 raft_consensus.cc:2454] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d842bbb29fae41959d4d1b0f21d1253f in term 1.
I20260619 02:34:49.211789 12199 raft_consensus.cc:2454] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d842bbb29fae41959d4d1b0f21d1253f in term 1.
I20260619 02:34:49.212133 12399 leader_election.cc:302] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba04ea6b37bd4193841d345f512cdf21, d842bbb29fae41959d4d1b0f21d1253f; no voters:
I20260619 02:34:49.212363 12534 raft_consensus.cc:2781] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:49.212627 12534 raft_consensus.cc:686] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 1 LEADER]: Becoming Leader. State: Replica: d842bbb29fae41959d4d1b0f21d1253f, State: Running, Role: LEADER
I20260619 02:34:49.212989 12534 consensus_queue.cc:237] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } }
I20260619 02:34:49.216174 12054 catalog_manager.cc:5495] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f reported cstate change: term changed from 0 to 1, leader changed from <none> to d842bbb29fae41959d4d1b0f21d1253f (127.7.194.195). New cstate: current_term: 1 leader_uuid: "d842bbb29fae41959d4d1b0f21d1253f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d842bbb29fae41959d4d1b0f21d1253f" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 } health_report { overall_health: UNKNOWN } } }
I20260619 02:34:49.244156 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.240420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.7.194.195:42031)
0619 02:34:49.240562 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.240566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.240581 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:34:49.240746 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.240748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.240806 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.240935 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:49.241176 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.241748 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.242608 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:49.242833 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:49.242942 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:49.243295 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:49.243743 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:49.243884 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.243925 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.243985 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:34:49.250437 12332 raft_consensus.cc:1261] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [term 1 FOLLOWER]: Refusing update from remote peer d842bbb29fae41959d4d1b0f21d1253f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:49.250437 12199 raft_consensus.cc:1261] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [term 1 FOLLOWER]: Refusing update from remote peer d842bbb29fae41959d4d1b0f21d1253f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:49.251200 12538 consensus_queue.cc:1035] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba04ea6b37bd4193841d345f512cdf21" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:49.251406 12534 consensus_queue.cc:1035] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [LEADER]: Connected to new peer: Peer: permanent_uuid: "576e7ac54cf546a78b461c9526008c15" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:49.269402 12055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:49.269640 12055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:49.272218 12055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bd3950cc4f5464db017e7d6690f986c: Sending DeleteTablet for 3 replicas of tablet 686eec692a314f7cae8e2fa63050ec97
I20260619 02:34:49.273072 12312 tablet_service.cc:1514] Processing DeleteTablet for tablet 686eec692a314f7cae8e2fa63050ec97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:49 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:39444
I20260619 02:34:49.273103 12445 tablet_service.cc:1514] Processing DeleteTablet for tablet 686eec692a314f7cae8e2fa63050ec97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:49 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:44790
I20260619 02:34:49.273587 12551 tablet_replica.cc:330] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: stopping tablet replica
I20260619 02:34:49.273963 12551 raft_consensus.cc:2229] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:49.274369 12551 raft_consensus.cc:2258] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:49.274317 12179 tablet_service.cc:1514] Processing DeleteTablet for tablet 686eec692a314f7cae8e2fa63050ec97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:49 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:49708
I20260619 02:34:49.274797 12553 tablet_replica.cc:330] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15: stopping tablet replica
I20260619 02:34:49.275012 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 12117
I20260619 02:34:49.275123 12553 raft_consensus.cc:2229] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:49.275404 12553 raft_consensus.cc:2258] T 686eec692a314f7cae8e2fa63050ec97 P 576e7ac54cf546a78b461c9526008c15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:49.277474 12551 ts_tablet_manager.cc:1905] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:49.280436 12551 ts_tablet_manager.cc:1918] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:49.280571 12551 log.cc:1196] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/686eec692a314f7cae8e2fa63050ec97
I20260619 02:34:49.280921 12551 ts_tablet_manager.cc:1939] T 686eec692a314f7cae8e2fa63050ec97 P ba04ea6b37bd4193841d345f512cdf21: Deleting consensus metadata
I20260619 02:34:49.281466 12552 tablet_replica.cc:330] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: stopping tablet replica
I20260619 02:34:49.281916 12552 raft_consensus.cc:2229] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:34:49.282337 12040 catalog_manager.cc:4841] TS ba04ea6b37bd4193841d345f512cdf21 (127.7.194.194:46379): tablet 686eec692a314f7cae8e2fa63050ec97 (table test-table [id=174cfb24848b45078273a2935b747cb2]) successfully deleted
I20260619 02:34:49.282449 12552 raft_consensus.cc:2258] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:34:49.283154 12042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:42547 (error 108)
I20260619 02:34:49.283977 12552 ts_tablet_manager.cc:1905] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:49.284009 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 12250
I20260619 02:34:49.287873 12552 ts_tablet_manager.cc:1918] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:49.288038 12552 log.cc:1196] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/686eec692a314f7cae8e2fa63050ec97
I20260619 02:34:49.288450 12552 ts_tablet_manager.cc:1939] T 686eec692a314f7cae8e2fa63050ec97 P d842bbb29fae41959d4d1b0f21d1253f: Deleting consensus metadata
I20260619 02:34:49.289851 12039 catalog_manager.cc:4841] TS d842bbb29fae41959d4d1b0f21d1253f (127.7.194.195:42031): tablet 686eec692a314f7cae8e2fa63050ec97 (table test-table [id=174cfb24848b45078273a2935b747cb2]) successfully deleted
I20260619 02:34:49.291085 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.290696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:42547 (local address 127.0.0.1:49724)
0619 02:34:49.290905 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:49.290981 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:42547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260619 02:34:49.291358 12042 catalog_manager.cc:4584] TS 576e7ac54cf546a78b461c9526008c15 (127.7.194.193:42547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 686eec692a314f7cae8e2fa63050ec97: Network error: Client connection negotiation failed: client connection to 127.7.194.193:42547: connect: Connection refused (error 111)
I20260619 02:34:49.292798 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 12383
I20260619 02:34:49.298403 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 12024
2026-06-19T02:34:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2380 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T02:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:49Z Disabled control of system clock
I20260619 02:34:49.330091 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:46863
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:44415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:34:49.446484 12558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:49.446784 12558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:49.446825 12558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:49.451115 12558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:49.451195 12558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:49.451215 12558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:49.451264 12558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:49.451306 12558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:49.456597 12558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:46863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:46863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:49.457782 12558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:49.459264 12558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:49.468025 12566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:49.468025 12563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:49.468067 12564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:49.468254 12558 server_base.cc:1034] running on GCE node
I20260619 02:34:49.468987 12558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:49.470263 12558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:49.471462 12558 hybrid_clock.cc:648] HybridClock initialized: now 1781836489471443 us; error 32 us; skew 500 ppm
I20260619 02:34:49.473395 12558 webserver.cc:460] Webserver started at http://127.7.194.254:34099/ using document root <none> and password file <none>
I20260619 02:34:49.474038 12558 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:49.474133 12558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:49.474364 12558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:49.476948 12558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "9febe19253a5486f866b011e28f54fcc"
format_stamp: "Formatted at 2026-06-19 02:34:49 on dist-test-slave-9tkz"
server_key: "608d462baa76636a0121aadbea48695f"
server_key_iv: "9e83423a3ed861942d890a22e827c8a6"
server_key_version: "encryptionkey@0"
I20260619 02:34:49.477685 12558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "9febe19253a5486f866b011e28f54fcc"
format_stamp: "Formatted at 2026-06-19 02:34:49 on dist-test-slave-9tkz"
server_key: "608d462baa76636a0121aadbea48695f"
server_key_iv: "9e83423a3ed861942d890a22e827c8a6"
server_key_version: "encryptionkey@0"
I20260619 02:34:49.481722 12558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:49.484126 12572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:49.485378 12558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:34:49.485504 12558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "9febe19253a5486f866b011e28f54fcc"
format_stamp: "Formatted at 2026-06-19 02:34:49 on dist-test-slave-9tkz"
server_key: "608d462baa76636a0121aadbea48695f"
server_key_iv: "9e83423a3ed861942d890a22e827c8a6"
server_key_version: "encryptionkey@0"
I20260619 02:34:49.485638 12558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:49.490244 12558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:49.490867 12558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:49.491052 12558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:49.497985 12558 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:46863
I20260619 02:34:49.498004 12624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:46863 every 8 connection(s)
I20260619 02:34:49.498945 12558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:49.502220 12625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:49.506467 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 12558
I20260619 02:34:49.506603 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:49.506896 7947 external_mini_cluster.cc:1428] Setting key 4aa76c01805c49402b0b80f1c0624375
I20260619 02:34:49.508553 12625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc: Bootstrap starting.
I20260619 02:34:49.511464 12625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:49.512471 12625 log.cc:824] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:49.513442 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.508494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.7.194.254:46863)
0619 02:34:49.509241 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.509269 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.509312 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:34:49.510238 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.510263 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.510737 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.511114 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:49.511925 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:49.511931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:49.511946 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:49.511956 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:49.512020 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:49.512040 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:49.512177 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.512260 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.512521 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":160,"threads_started":1}
I20260619 02:34:49.514643 12625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc: No bootstrap required, opened a new log
I20260619 02:34:49.518352 12625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9febe19253a5486f866b011e28f54fcc" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46863 } }
I20260619 02:34:49.518640 12625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:49.518715 12625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9febe19253a5486f866b011e28f54fcc, State: Initialized, Role: FOLLOWER
I20260619 02:34:49.519172 12625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9febe19253a5486f866b011e28f54fcc" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46863 } }
I20260619 02:34:49.519300 12625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:49.519340 12625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:49.519408 12625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:49.520311 12625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9febe19253a5486f866b011e28f54fcc" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46863 } }
I20260619 02:34:49.520609 12625 leader_election.cc:302] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9febe19253a5486f866b011e28f54fcc; no voters:
I20260619 02:34:49.520872 12625 leader_election.cc:288] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:49.521020 12630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:49.521296 12630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [term 1 LEADER]: Becoming Leader. State: Replica: 9febe19253a5486f866b011e28f54fcc, State: Running, Role: LEADER
I20260619 02:34:49.521684 12630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9febe19253a5486f866b011e28f54fcc" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46863 } }
I20260619 02:34:49.522351 12625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:49.523559 12632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9febe19253a5486f866b011e28f54fcc. Latest consensus state: current_term: 1 leader_uuid: "9febe19253a5486f866b011e28f54fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9febe19253a5486f866b011e28f54fcc" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46863 } } }
I20260619 02:34:49.523731 12632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:49.523577 12631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9febe19253a5486f866b011e28f54fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9febe19253a5486f866b011e28f54fcc" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46863 } } }
I20260619 02:34:49.523818 12631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:49.524222 12640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:49.527150 12640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:49.532406 12640 catalog_manager.cc:1347] Generated new cluster ID: 166abe1ffd5b4698abda70a7487082c0
I20260619 02:34:49.532490 12640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:49.548974 12640 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:49.549362 12640 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:49.555718 12640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc: Generated new TSK 0
I20260619 02:34:49.556612 12640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:34:49.570200 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46863
--builtin_ntp_servers=127.7.194.212:44415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:34:49.685688 12649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:49.685947 12649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:49.686021 12649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:49.690207 12649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:49.690258 12649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:49.690371 12649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:49.695180 12649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:49.696309 12649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:49.697798 12649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:49.707755 12657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:49.707729 12654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:49.707686 12649 server_base.cc:1034] running on GCE node
W20260619 02:34:49.707675 12655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:49.708410 12649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:49.709059 12649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:49.710219 12649 hybrid_clock.cc:648] HybridClock initialized: now 1781836489710197 us; error 34 us; skew 500 ppm
I20260619 02:34:49.712218 12649 webserver.cc:460] Webserver started at http://127.7.194.193:38855/ using document root <none> and password file <none>
I20260619 02:34:49.712857 12649 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:49.712922 12649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:49.713260 12649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:49.715543 12649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "cfc516eca4cc4c458de0351dfa3ba204"
format_stamp: "Formatted at 2026-06-19 02:34:49 on dist-test-slave-9tkz"
server_key: "dfba196a06c937ae1315ea84205dcba2"
server_key_iv: "6d3a9ea4a7dfbb61b204100e4f69c647"
server_key_version: "encryptionkey@0"
I20260619 02:34:49.716156 12649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "cfc516eca4cc4c458de0351dfa3ba204"
format_stamp: "Formatted at 2026-06-19 02:34:49 on dist-test-slave-9tkz"
server_key: "dfba196a06c937ae1315ea84205dcba2"
server_key_iv: "6d3a9ea4a7dfbb61b204100e4f69c647"
server_key_version: "encryptionkey@0"
I20260619 02:34:49.720568 12649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:34:49.723356 12663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:49.724709 12649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:34:49.724828 12649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "cfc516eca4cc4c458de0351dfa3ba204"
format_stamp: "Formatted at 2026-06-19 02:34:49 on dist-test-slave-9tkz"
server_key: "dfba196a06c937ae1315ea84205dcba2"
server_key_iv: "6d3a9ea4a7dfbb61b204100e4f69c647"
server_key_version: "encryptionkey@0"
I20260619 02:34:49.724949 12649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:49.729786 12649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:49.730441 12649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:49.730628 12649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:49.731184 12649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:49.732103 12649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:49.732148 12649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:49.732213 12649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:49.732241 12649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:49.740437 12649 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:46605
I20260619 02:34:49.740506 12776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:46605 every 8 connection(s)
I20260619 02:34:49.741405 12649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:49.746915 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 12649
I20260619 02:34:49.747051 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:49.747398 7947 external_mini_cluster.cc:1428] Setting key f59033402ce31d84393fc0ae0a77e188
I20260619 02:34:49.747705 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.743336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:49679 (local address 127.7.194.254:46863)
0619 02:34:49.743504 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.743509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.744603 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 02:34:49.745897 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.745901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.745942 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.746025 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:49.747227 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:49.747230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:49.747233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:49.747248 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:49.747269 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:49.747277 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:49.747359 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.747535 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.747583 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:34:49.748883 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.743768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:46863 (local address 127.7.194.193:49679)
0619 02:34:49.744431 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:49.744472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:49.745685 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:49.746198 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:49.746205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:49.746295 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:49.746894 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:34:49.746896 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:49.746954 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:34:49.747053 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:49.747366 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:49.747374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:49.747380 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:34:49.747563 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 02:34:49.747818 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260619 02:34:49.749732 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46863
--builtin_ntp_servers=127.7.194.212:44415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:34:49.750176 12777 heartbeater.cc:347] Connected to a master server at 127.7.194.254:46863
I20260619 02:34:49.750406 12777 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:49.750780 12777 heartbeater.cc:511] Master 127.7.194.254:46863 requested a full tablet report, sending...
I20260619 02:34:49.752462 12589 ts_manager.cc:194] Registered new tserver with Master: cfc516eca4cc4c458de0351dfa3ba204 (127.7.194.193:46605)
W20260619 02:34:49.864658 12780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:49.864993 12780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:49.865070 12780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:49.869374 12780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:49.869477 12780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:49.869632 12780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:49.874845 12780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:49.876088 12780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:49.877600 12780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:49.887665 12785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:49.887770 12788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:49.887881 12786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:49.888455 12780 server_base.cc:1034] running on GCE node
I20260619 02:34:49.888909 12780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:49.889675 12780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:49.890865 12780 hybrid_clock.cc:648] HybridClock initialized: now 1781836489890845 us; error 42 us; skew 500 ppm
I20260619 02:34:49.892866 12780 webserver.cc:460] Webserver started at http://127.7.194.194:37211/ using document root <none> and password file <none>
I20260619 02:34:49.893473 12780 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:49.893596 12780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:49.893842 12780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:49.895969 12780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "2c0d86e544d34e4c8744a0bc6104bf3a"
format_stamp: "Formatted at 2026-06-19 02:34:49 on dist-test-slave-9tkz"
server_key: "c2da0ece9480070c78955762b86a631a"
server_key_iv: "fa161348b16547c5214af4d37277f29a"
server_key_version: "encryptionkey@0"
I20260619 02:34:49.896546 12780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "2c0d86e544d34e4c8744a0bc6104bf3a"
format_stamp: "Formatted at 2026-06-19 02:34:49 on dist-test-slave-9tkz"
server_key: "c2da0ece9480070c78955762b86a631a"
server_key_iv: "fa161348b16547c5214af4d37277f29a"
server_key_version: "encryptionkey@0"
I20260619 02:34:49.900417 12780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 02:34:49.902990 12794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:49.904423 12780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:34:49.904556 12780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "2c0d86e544d34e4c8744a0bc6104bf3a"
format_stamp: "Formatted at 2026-06-19 02:34:49 on dist-test-slave-9tkz"
server_key: "c2da0ece9480070c78955762b86a631a"
server_key_iv: "fa161348b16547c5214af4d37277f29a"
server_key_version: "encryptionkey@0"
I20260619 02:34:49.904706 12780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:49.909518 12780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:49.910183 12780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:49.910377 12780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:49.910908 12780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:49.911804 12780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:49.911873 12780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:49.911942 12780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:49.911993 12780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:49.920890 12780 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:40085
I20260619 02:34:49.920915 12907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:40085 every 8 connection(s)
I20260619 02:34:49.921914 12780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:49.927791 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 12780
I20260619 02:34:49.927898 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:49.928297 7947 external_mini_cluster.cc:1428] Setting key e8f024e4beaa2d2652bf7d4892404930
I20260619 02:34:49.928797 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.923998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:59013 (local address 127.7.194.254:46863)
0619 02:34:49.924173 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:49.924178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:49.925382 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 02:34:49.926748 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:49.926750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:49.926821 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:49.926902 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:49.928260 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:49.928264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:49.928266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:49.928280 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:49.928308 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:49.928321 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:49.928431 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:49.928584 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 02:34:49.928641 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:34:49.929798 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:49.924552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:46863 (local address 127.7.194.194:59013)
0619 02:34:49.925218 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:49.925261 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:49.926524 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:49.927063 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:49.927069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:49.927161 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:49.927818 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:34:49.927819 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:49.927991 (+ 172us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:34:49.928103 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:49.928423 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:49.928431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:49.928437 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:34:49.928595 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 02:34:49.928798 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260619 02:34:49.930502 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46863
--builtin_ntp_servers=127.7.194.212:44415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:34:49.931124 12908 heartbeater.cc:347] Connected to a master server at 127.7.194.254:46863
I20260619 02:34:49.931378 12908 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:49.931838 12908 heartbeater.cc:511] Master 127.7.194.254:46863 requested a full tablet report, sending...
I20260619 02:34:49.933046 12589 ts_manager.cc:194] Registered new tserver with Master: 2c0d86e544d34e4c8744a0bc6104bf3a (127.7.194.194:40085)
W20260619 02:34:50.046685 12911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:50.046972 12911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:50.047014 12911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:50.051240 12911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:50.051309 12911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:50.051406 12911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:50.056484 12911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:50.057806 12911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:50.059280 12911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:50.069147 12916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:50.069144 12917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:50.069337 12911 server_base.cc:1034] running on GCE node
W20260619 02:34:50.069235 12919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:50.069900 12911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:50.070575 12911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:50.071766 12911 hybrid_clock.cc:648] HybridClock initialized: now 1781836490071739 us; error 39 us; skew 500 ppm
I20260619 02:34:50.073722 12911 webserver.cc:460] Webserver started at http://127.7.194.195:42465/ using document root <none> and password file <none>
I20260619 02:34:50.074323 12911 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:50.074414 12911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:50.074647 12911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:50.076799 12911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6"
format_stamp: "Formatted at 2026-06-19 02:34:50 on dist-test-slave-9tkz"
server_key: "4a2e65e621be5f6568780f6ee8c03a66"
server_key_iv: "399f8c3d3486cece264374225a4fac4b"
server_key_version: "encryptionkey@0"
I20260619 02:34:50.077390 12911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6"
format_stamp: "Formatted at 2026-06-19 02:34:50 on dist-test-slave-9tkz"
server_key: "4a2e65e621be5f6568780f6ee8c03a66"
server_key_iv: "399f8c3d3486cece264374225a4fac4b"
server_key_version: "encryptionkey@0"
I20260619 02:34:50.081101 12911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:34:50.083809 12925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:50.085105 12911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:34:50.085245 12911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6"
format_stamp: "Formatted at 2026-06-19 02:34:50 on dist-test-slave-9tkz"
server_key: "4a2e65e621be5f6568780f6ee8c03a66"
server_key_iv: "399f8c3d3486cece264374225a4fac4b"
server_key_version: "encryptionkey@0"
I20260619 02:34:50.085364 12911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:50.090049 12911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:50.090682 12911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:50.090864 12911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:50.091419 12911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:50.092332 12911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:50.092402 12911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:50.092471 12911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:50.092521 12911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:50.101378 12911 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:40339
I20260619 02:34:50.101399 13038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:40339 every 8 connection(s)
I20260619 02:34:50.102383 12911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:50.107434 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 12911
I20260619 02:34:50.107537 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:50.107968 7947 external_mini_cluster.cc:1428] Setting key 60044fcc0b94754f42522544c2ea104c
I20260619 02:34:50.109117 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.104322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:57777 (local address 127.7.194.254:46863)
0619 02:34:50.104472 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.104476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.105686 (+ 1210us) server_negotiation.cc:407] Connection header received
0619 02:34:50.107059 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.107062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.107109 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.107197 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.108573 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.108577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.108580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.108633 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.108659 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.108671 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.108783 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.108955 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.109001 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:34:50.110323 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.104749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:46863 (local address 127.7.194.195:57777)
0619 02:34:50.105439 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:50.105482 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:50.106837 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:50.107364 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:50.107371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:50.107468 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:50.108251 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:34:50.108253 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:50.108320 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:34:50.108412 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:50.108791 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:50.108799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:50.108806 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:34:50.108993 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 02:34:50.109301 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260619 02:34:50.111430 13039 heartbeater.cc:347] Connected to a master server at 127.7.194.254:46863
I20260619 02:34:50.111650 13039 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:50.112042 13039 heartbeater.cc:511] Master 127.7.194.254:46863 requested a full tablet report, sending...
I20260619 02:34:50.113040 12589 ts_manager.cc:194] Registered new tserver with Master: 30a7b6b34a7d46edb84b9f1eb8319ed6 (127.7.194.195:40339)
I20260619 02:34:50.122298 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:34:50.126257 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.124496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.7.194.254:46863)
0619 02:34:50.124662 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.124673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.124883 (+ 210us) server_negotiation.cc:407] Connection header received
0619 02:34:50.124955 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.124957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.124995 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.125071 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.125881 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.125884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.125886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.125897 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.125916 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.125924 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.125994 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.126096 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.126134 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:34:50.129333 12589 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:34:50.132545 12589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:50.144500 13050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.141191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:40085 (local address 127.0.0.1:44670)
0619 02:34:50.141785 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:50.141842 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:50.141955 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:50.142501 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:50.142507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:50.142588 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:50.143243 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:34:50.143247 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:50.143270 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:34:50.143346 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:50.144163 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:50.144171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:50.144177 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:34:50.144283 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:34:50.144352 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":128,"threads_started":1}
I20260619 02:34:50.144897 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.141832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46490 (local address 127.7.194.195:40339)
0619 02:34:50.142335 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.142350 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.142685 (+ 335us) server_negotiation.cc:407] Connection header received
0619 02:34:50.142870 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.142878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.143092 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.143215 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.143969 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.143976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.143987 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.143998 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.144043 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.144092 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.144214 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.144440 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.144518 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":167,"threads_started":1}
I20260619 02:34:50.144968 13053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.142237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:40339 (local address 127.0.0.1:46490)
0619 02:34:50.142553 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:50.142568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:50.142743 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:50.143262 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:50.143265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:50.143280 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:50.143808 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:34:50.143810 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:50.143821 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:34:50.143852 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:50.144264 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:50.144273 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:50.144280 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:34:50.144413 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 02:34:50.144455 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
I20260619 02:34:50.145016 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.141377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.7.194.194:40085)
0619 02:34:50.141981 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.141993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.142020 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:34:50.142085 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.142094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.142328 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.142903 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.143953 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.143958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.143970 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.143979 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.144018 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.144050 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.144713 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.144765 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.144839 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":114,"threads_started":1}
I20260619 02:34:50.145123 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.141995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46605 (local address 127.0.0.1:45848)
0619 02:34:50.142491 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:50.142506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:50.142588 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:50.143984 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:50.143987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:50.144003 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:50.144450 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:34:50.144452 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:50.144461 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:34:50.144487 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:50.144877 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:50.144880 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:50.144883 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:34:50.144959 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:34:50.144997 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":103,"threads_started":1}
I20260619 02:34:50.145535 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.142747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.7.194.193:46605)
0619 02:34:50.143417 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.143430 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.143453 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:34:50.143504 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.143508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.143670 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.143807 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.144607 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.144617 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.144629 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.144647 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.144695 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.144729 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.144823 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.145257 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.145332 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":119,"threads_started":1}
I20260619 02:34:50.148033 12973 tablet_service.cc:1467] Processing CreateTablet for tablet a6e89674b0d04c0d8dacc353f5118e91 (DEFAULT_TABLE table=test-table [id=251d60d95f264dcab986a3c06ad8a810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:50.148109 12711 tablet_service.cc:1467] Processing CreateTablet for tablet a6e89674b0d04c0d8dacc353f5118e91 (DEFAULT_TABLE table=test-table [id=251d60d95f264dcab986a3c06ad8a810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:50.148106 12842 tablet_service.cc:1467] Processing CreateTablet for tablet a6e89674b0d04c0d8dacc353f5118e91 (DEFAULT_TABLE table=test-table [id=251d60d95f264dcab986a3c06ad8a810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:50.149093 12973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e89674b0d04c0d8dacc353f5118e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:50.149150 12711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e89674b0d04c0d8dacc353f5118e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:50.149150 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e89674b0d04c0d8dacc353f5118e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:50.155776 13056 tablet_bootstrap.cc:492] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: Bootstrap starting.
I20260619 02:34:50.157176 13058 tablet_bootstrap.cc:492] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: Bootstrap starting.
I20260619 02:34:50.157176 13057 tablet_bootstrap.cc:492] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204: Bootstrap starting.
I20260619 02:34:50.158306 13056 tablet_bootstrap.cc:654] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:50.159181 13056 log.cc:824] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:50.159502 13057 tablet_bootstrap.cc:654] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:50.159502 13058 tablet_bootstrap.cc:654] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:50.160365 13057 log.cc:824] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:50.160367 13058 log.cc:824] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:50.161190 13056 tablet_bootstrap.cc:492] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: No bootstrap required, opened a new log
I20260619 02:34:50.161392 13056 ts_tablet_manager.cc:1397] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:34:50.162513 13058 tablet_bootstrap.cc:492] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: No bootstrap required, opened a new log
I20260619 02:34:50.162513 13057 tablet_bootstrap.cc:492] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204: No bootstrap required, opened a new log
I20260619 02:34:50.162720 13057 ts_tablet_manager.cc:1397] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:34:50.162720 13058 ts_tablet_manager.cc:1397] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:34:50.165385 13056 raft_consensus.cc:348] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } }
I20260619 02:34:50.165707 13056 raft_consensus.cc:374] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:50.165783 13056 raft_consensus.cc:729] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a7b6b34a7d46edb84b9f1eb8319ed6, State: Initialized, Role: FOLLOWER
I20260619 02:34:50.166261 13056 consensus_queue.cc:260] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } }
I20260619 02:34:50.166960 13039 heartbeater.cc:503] Master 127.7.194.254:46863 was elected leader, sending a full tablet report...
I20260619 02:34:50.167354 13056 ts_tablet_manager.cc:1428] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:34:50.167562 13058 raft_consensus.cc:348] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } }
I20260619 02:34:50.167562 13057 raft_consensus.cc:348] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } }
I20260619 02:34:50.167860 13058 raft_consensus.cc:374] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:50.167860 13057 raft_consensus.cc:374] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:50.167932 13057 raft_consensus.cc:729] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc516eca4cc4c458de0351dfa3ba204, State: Initialized, Role: FOLLOWER
I20260619 02:34:50.167954 13058 raft_consensus.cc:729] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0d86e544d34e4c8744a0bc6104bf3a, State: Initialized, Role: FOLLOWER
I20260619 02:34:50.168433 13057 consensus_queue.cc:260] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } }
I20260619 02:34:50.168433 13058 consensus_queue.cc:260] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } }
I20260619 02:34:50.169401 12777 heartbeater.cc:503] Master 127.7.194.254:46863 was elected leader, sending a full tablet report...
I20260619 02:34:50.169443 12908 heartbeater.cc:503] Master 127.7.194.254:46863 was elected leader, sending a full tablet report...
I20260619 02:34:50.169519 13057 ts_tablet_manager.cc:1428] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 02:34:50.170109 13058 ts_tablet_manager.cc:1428] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
W20260619 02:34:50.172933 12909 tablet.cc:2369] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:34:50.242635 12778 tablet.cc:2369] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:34:50.286273 13062 raft_consensus.cc:482] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:50.286550 13062 raft_consensus.cc:504] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } }
I20260619 02:34:50.287786 13062 leader_election.cc:288] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c0d86e544d34e4c8744a0bc6104bf3a (127.7.194.194:40085), cfc516eca4cc4c458de0351dfa3ba204 (127.7.194.193:46605)
I20260619 02:34:50.290042 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.288150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:40085 (local address 127.7.194.195:56495)
0619 02:34:50.288337 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:50.288355 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:50.288461 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:50.288800 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:50.288804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:50.288820 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:50.289415 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:34:50.289418 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:50.289436 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:34:50.289472 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:50.289785 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:50.289788 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:50.289792 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:34:50.289873 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:34:50.289911 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 02:34:50.290042 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.288263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:56495 (local address 127.7.194.194:40085)
0619 02:34:50.288405 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.288412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.288428 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:34:50.288605 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.288608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.288647 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.288752 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.289605 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.289608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.289611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.289622 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.289683 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.289694 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.289819 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.289873 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.289911 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:34:50.290375 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.288636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:38997 (local address 127.7.194.193:46605)
0619 02:34:50.288772 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.288777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.288956 (+ 179us) server_negotiation.cc:407] Connection header received
0619 02:34:50.289063 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.289067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.289152 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.289223 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.289988 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.289992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.289994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.290006 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.290026 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.290037 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.290147 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.290203 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.290243 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:34:50.290392 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.288491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46605 (local address 127.7.194.195:38997)
0619 02:34:50.288868 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:50.288879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:50.288947 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:50.289272 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:50.289276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:50.289292 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:50.289821 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:34:50.289823 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:50.289835 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:34:50.289862 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:50.290124 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:50.290127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:50.290130 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:34:50.290211 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:34:50.290243 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":127,"threads_started":1}
I20260619 02:34:50.290876 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e89674b0d04c0d8dacc353f5118e91" candidate_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" is_pre_election: true
I20260619 02:34:50.290985 12731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e89674b0d04c0d8dacc353f5118e91" candidate_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc516eca4cc4c458de0351dfa3ba204" is_pre_election: true
I20260619 02:34:50.291242 12862 raft_consensus.cc:2454] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a7b6b34a7d46edb84b9f1eb8319ed6 in term 0.
I20260619 02:34:50.291352 12731 raft_consensus.cc:2454] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a7b6b34a7d46edb84b9f1eb8319ed6 in term 0.
I20260619 02:34:50.291785 12929 leader_election.cc:302] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c0d86e544d34e4c8744a0bc6104bf3a, 30a7b6b34a7d46edb84b9f1eb8319ed6; no voters:
I20260619 02:34:50.292018 13062 raft_consensus.cc:2781] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:50.292114 13062 raft_consensus.cc:482] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:50.292186 13062 raft_consensus.cc:3037] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:50.293311 13062 raft_consensus.cc:504] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } }
I20260619 02:34:50.293723 13062 leader_election.cc:288] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [CANDIDATE]: Term 1 election: Requested vote from peers 2c0d86e544d34e4c8744a0bc6104bf3a (127.7.194.194:40085), cfc516eca4cc4c458de0351dfa3ba204 (127.7.194.193:46605)
I20260619 02:34:50.294103 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e89674b0d04c0d8dacc353f5118e91" candidate_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a"
I20260619 02:34:50.294145 12731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e89674b0d04c0d8dacc353f5118e91" candidate_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc516eca4cc4c458de0351dfa3ba204"
I20260619 02:34:50.294265 12862 raft_consensus.cc:3037] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:50.294318 12731 raft_consensus.cc:3037] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:50.295395 12731 raft_consensus.cc:2454] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a7b6b34a7d46edb84b9f1eb8319ed6 in term 1.
I20260619 02:34:50.295404 12862 raft_consensus.cc:2454] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a7b6b34a7d46edb84b9f1eb8319ed6 in term 1.
I20260619 02:34:50.295687 12929 leader_election.cc:302] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a7b6b34a7d46edb84b9f1eb8319ed6, cfc516eca4cc4c458de0351dfa3ba204; no voters:
I20260619 02:34:50.295934 13062 raft_consensus.cc:2781] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:50.296217 13062 raft_consensus.cc:686] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 1 LEADER]: Becoming Leader. State: Replica: 30a7b6b34a7d46edb84b9f1eb8319ed6, State: Running, Role: LEADER
I20260619 02:34:50.296568 13062 consensus_queue.cc:237] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } }
I20260619 02:34:50.300465 12588 catalog_manager.cc:5495] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a7b6b34a7d46edb84b9f1eb8319ed6 (127.7.194.195). New cstate: current_term: 1 leader_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a7b6b34a7d46edb84b9f1eb8319ed6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40339 } health_report { overall_health: HEALTHY } } }
I20260619 02:34:50.339064 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.337431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.7.194.195:40339)
0619 02:34:50.337625 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.337629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.337643 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:50.337795 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.337798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.337863 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.337934 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.338602 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.338605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.338607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.338618 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.338638 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.338647 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.338731 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.338850 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.338885 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:34:50.345412 12862 raft_consensus.cc:1261] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [term 1 FOLLOWER]: Refusing update from remote peer 30a7b6b34a7d46edb84b9f1eb8319ed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:50.345412 12731 raft_consensus.cc:1261] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [term 1 FOLLOWER]: Refusing update from remote peer 30a7b6b34a7d46edb84b9f1eb8319ed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:50.346151 13062 consensus_queue.cc:1035] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0d86e544d34e4c8744a0bc6104bf3a" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:50.346386 13066 consensus_queue.cc:1035] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc516eca4cc4c458de0351dfa3ba204" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:34:50.353345 13040 tablet.cc:2369] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:34:50.363492 12587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:50.363701 12587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:50.366168 12587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9febe19253a5486f866b011e28f54fcc: Sending DeleteTablet for 3 replicas of tablet a6e89674b0d04c0d8dacc353f5118e91
I20260619 02:34:50.367015 12973 tablet_service.cc:1514] Processing DeleteTablet for tablet a6e89674b0d04c0d8dacc353f5118e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:50 UTC) from {username='slave'} at 127.0.0.1:46490
I20260619 02:34:50.367043 12711 tablet_service.cc:1514] Processing DeleteTablet for tablet a6e89674b0d04c0d8dacc353f5118e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:50 UTC) from {username='slave'} at 127.0.0.1:45848
I20260619 02:34:50.367080 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet a6e89674b0d04c0d8dacc353f5118e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:50 UTC) from {username='slave'} at 127.0.0.1:44670
I20260619 02:34:50.367553 13081 tablet_replica.cc:330] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: stopping tablet replica
I20260619 02:34:50.367597 13080 tablet_replica.cc:330] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204: stopping tablet replica
I20260619 02:34:50.367975 13080 raft_consensus.cc:2229] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:50.368319 13080 raft_consensus.cc:2258] T a6e89674b0d04c0d8dacc353f5118e91 P cfc516eca4cc4c458de0351dfa3ba204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:50.368673 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 12649
I20260619 02:34:50.368676 13079 tablet_replica.cc:330] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: stopping tablet replica
I20260619 02:34:50.368697 13081 raft_consensus.cc:2229] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:50.368968 13079 raft_consensus.cc:2229] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:34:50.369328 13079 raft_consensus.cc:2258] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:50.370676 13081 raft_consensus.cc:2258] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:50.371075 13079 ts_tablet_manager.cc:1905] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:50.372144 13081 ts_tablet_manager.cc:1905] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:50.373320 13079 ts_tablet_manager.cc:1918] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:50.373433 13079 log.cc:1196] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/a6e89674b0d04c0d8dacc353f5118e91
I20260619 02:34:50.373850 13079 ts_tablet_manager.cc:1939] T a6e89674b0d04c0d8dacc353f5118e91 P 30a7b6b34a7d46edb84b9f1eb8319ed6: Deleting consensus metadata
I20260619 02:34:50.374927 13081 ts_tablet_manager.cc:1918] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:50.375098 13081 log.cc:1196] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/a6e89674b0d04c0d8dacc353f5118e91
I20260619 02:34:50.375226 12573 catalog_manager.cc:4841] TS 30a7b6b34a7d46edb84b9f1eb8319ed6 (127.7.194.195:40339): tablet a6e89674b0d04c0d8dacc353f5118e91 (table test-table [id=251d60d95f264dcab986a3c06ad8a810]) successfully deleted
I20260619 02:34:50.375530 13081 ts_tablet_manager.cc:1939] T a6e89674b0d04c0d8dacc353f5118e91 P 2c0d86e544d34e4c8744a0bc6104bf3a: Deleting consensus metadata
I20260619 02:34:50.376781 12576 catalog_manager.cc:4841] TS 2c0d86e544d34e4c8744a0bc6104bf3a (127.7.194.194:40085): tablet a6e89674b0d04c0d8dacc353f5118e91 (table test-table [id=251d60d95f264dcab986a3c06ad8a810]) successfully deleted
W20260619 02:34:50.378450 12576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:46605 (error 108)
I20260619 02:34:50.378633 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 12780
I20260619 02:34:50.380525 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.380202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46605 (local address 127.0.0.1:45858)
0619 02:34:50.380350 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:50.380432 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:46605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:34:50.380713 12576 catalog_manager.cc:4584] TS cfc516eca4cc4c458de0351dfa3ba204 (127.7.194.193:46605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6e89674b0d04c0d8dacc353f5118e91: Network error: Client connection negotiation failed: client connection to 127.7.194.193:46605: connect: Connection refused (error 111)
I20260619 02:34:50.384379 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 12911
I20260619 02:34:50.391681 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 12558
2026-06-19T02:34:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1093 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 02:34:50.414638 7947 mini_oidc.cc:152] Starting JWKS server
I20260619 02:34:50.416246 7947 webserver.cc:460] Webserver started at https://127.0.0.1:32821/ using document root <none> and password file <none>
I20260619 02:34:50.416386 7947 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:34:50.416606 7947 webserver.cc:460] Webserver started at http://127.0.0.1:43337/ using document root <none> and password file <none>
2026-06-19T02:34:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:50Z Disabled control of system clock
I20260619 02:34:50.426188 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:37963
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:33793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:37963
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32821/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/testchainca.cert with env {}
W20260619 02:34:50.542655 13088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:50.542968 13088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:50.543010 13088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:50.547137 13088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:50.547210 13088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:50.547226 13088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:50.547243 13088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:50.547259 13088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:34:50.547276 13088 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:34:50.547290 13088 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32821/jwks.json
I20260619 02:34:50.552982 13088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:33793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:37963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:37963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32821/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:50.554191 13088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:50.555222 13088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:50.564334 13093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:50.564350 13096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:50.564289 13094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:50.564812 13088 server_base.cc:1034] running on GCE node
I20260619 02:34:50.565423 13088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:50.566658 13088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:50.567873 13088 hybrid_clock.cc:648] HybridClock initialized: now 1781836490567856 us; error 33 us; skew 500 ppm
I20260619 02:34:50.570067 13088 webserver.cc:460] Webserver started at http://127.7.194.254:42885/ using document root <none> and password file <none>
I20260619 02:34:50.570708 13088 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:50.570807 13088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:50.571079 13088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:50.573287 13088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "0c2d2f0e8d104976996856a1377f25b8"
format_stamp: "Formatted at 2026-06-19 02:34:50 on dist-test-slave-9tkz"
server_key: "9f60e720c903adaaaa7910f16a2586e3"
server_key_iv: "3a3928f8e2af9df98b8920372849f45e"
server_key_version: "encryptionkey@0"
I20260619 02:34:50.573952 13088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "0c2d2f0e8d104976996856a1377f25b8"
format_stamp: "Formatted at 2026-06-19 02:34:50 on dist-test-slave-9tkz"
server_key: "9f60e720c903adaaaa7910f16a2586e3"
server_key_iv: "3a3928f8e2af9df98b8920372849f45e"
server_key_version: "encryptionkey@0"
I20260619 02:34:50.577826 13088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:50.580372 13102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:50.581866 13088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:50.582002 13088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "0c2d2f0e8d104976996856a1377f25b8"
format_stamp: "Formatted at 2026-06-19 02:34:50 on dist-test-slave-9tkz"
server_key: "9f60e720c903adaaaa7910f16a2586e3"
server_key_iv: "3a3928f8e2af9df98b8920372849f45e"
server_key_version: "encryptionkey@0"
I20260619 02:34:50.582145 13088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:50.600577 13088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:50.601387 13088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:50.601666 13088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:50.608950 13088 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:37963
I20260619 02:34:50.608973 13154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:37963 every 8 connection(s)
I20260619 02:34:50.610059 13088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:50.612368 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 13088
I20260619 02:34:50.612471 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:50.612769 7947 external_mini_cluster.cc:1428] Setting key b54acd0ae329878080533adb400facc9
I20260619 02:34:50.613291 13155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:50.621026 13155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8: Bootstrap starting.
I20260619 02:34:50.622512 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.614209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.7.194.254:37963)
0619 02:34:50.614849 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.614865 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.614903 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:34:50.615557 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.615574 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.616047 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.616353 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.616846 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:50.617896 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:50.619092 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:50.619580 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:50.619706 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:50.620613 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.620618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.620632 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.621043 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.621102 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.621122 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.621260 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.621460 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.621762 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260619 02:34:50.623883 13155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:50.624806 13155 log.cc:824] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:50.625810 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:34:50.627034 13155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8: No bootstrap required, opened a new log
I20260619 02:34:50.630995 13155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2d2f0e8d104976996856a1377f25b8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37963 } }
I20260619 02:34:50.631239 13155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:50.631304 13155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2d2f0e8d104976996856a1377f25b8, State: Initialized, Role: FOLLOWER
I20260619 02:34:50.631733 13155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2d2f0e8d104976996856a1377f25b8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37963 } }
I20260619 02:34:50.631850 13155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:50.631918 13155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:50.632022 13155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:50.633044 13155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2d2f0e8d104976996856a1377f25b8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37963 } }
I20260619 02:34:50.633399 13155 leader_election.cc:302] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c2d2f0e8d104976996856a1377f25b8; no voters:
I20260619 02:34:50.633785 13155 leader_election.cc:288] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:50.634006 13160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:50.634251 13160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [term 1 LEADER]: Becoming Leader. State: Replica: 0c2d2f0e8d104976996856a1377f25b8, State: Running, Role: LEADER
I20260619 02:34:50.634591 13160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2d2f0e8d104976996856a1377f25b8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37963 } }
I20260619 02:34:50.635241 13155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:50.636451 13162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c2d2f0e8d104976996856a1377f25b8. Latest consensus state: current_term: 1 leader_uuid: "0c2d2f0e8d104976996856a1377f25b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2d2f0e8d104976996856a1377f25b8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37963 } } }
I20260619 02:34:50.636451 13161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c2d2f0e8d104976996856a1377f25b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2d2f0e8d104976996856a1377f25b8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37963 } } }
I20260619 02:34:50.636607 13162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:50.636607 13161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:50.637054 13169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:50.640049 13169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:50.646199 13169 catalog_manager.cc:1347] Generated new cluster ID: 9a7abe648cdb4ab998b5bba0a0508ca3
I20260619 02:34:50.646291 13169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:50.660671 13169 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:50.662056 13169 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:50.676837 13169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c2d2f0e8d104976996856a1377f25b8: Generated new TSK 0
I20260619 02:34:50.677879 13169 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:34:50.708662 13185 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 02:34:50.708808 13185 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:34:50.709475 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.691642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.7.194.254:37963)
0619 02:34:50.691805 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.691810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.691972 (+ 162us) server_negotiation.cc:407] Connection header received
0619 02:34:50.692202 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.692206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.692270 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.692388 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:34:50.692644 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:50.693478 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:50.694452 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:50.694782 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:50.694889 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:50.695242 (+ 353us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:34:50.708994 (+ 13752us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:34:50.709317 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58384: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 02:34:50.709594 13158 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58384: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 02:34:50.710824 13184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:34:50.691541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:37963 (local address 127.0.0.1:58384)
0619 02:34:50.691869 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:50.691884 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:50.692032 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:50.692416 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:50.692418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:50.692425 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:34:50.692540 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:50.692544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:50.693684 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:50.693688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:50.694329 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:50.694338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:50.694910 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:50.694913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:50.694987 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:50.695035 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:34:50.709258 (+ 14223us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:34:50.709514 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.194.254:37963: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":259,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.194.254:37963: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 02:34:50.711701 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 13088
2026-06-19T02:34:50Z chronyd exiting
I20260619 02:34:50.749529 7947 test_util.cc:175] -----------------------------------------------
I20260619 02:34:50.749691 7947 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781836464269074-7947-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 02:34:50.750777 7947 mini_oidc.cc:152] Starting JWKS server
I20260619 02:34:50.751665 7947 webserver.cc:460] Webserver started at https://127.0.0.1:41123/ using document root <none> and password file <none>
I20260619 02:34:50.751773 7947 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:34:50.752054 7947 webserver.cc:460] Webserver started at http://127.0.0.1:41925/ using document root <none> and password file <none>
2026-06-19T02:34:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:50Z Disabled control of system clock
I20260619 02:34:50.761385 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:40535
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:40535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41123/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/testchainca.cert with env {}
W20260619 02:34:50.874768 13191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:50.875069 13191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:50.875111 13191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:50.879282 13191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:50.879350 13191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:50.879366 13191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:50.879383 13191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:50.879398 13191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:34:50.879448 13191 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:34:50.879484 13191 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41123/jwks.json
I20260619 02:34:50.885185 13191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:40535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:40535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41123/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:50.886400 13191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:50.887516 13191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:50.896919 13196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:50.896922 13197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:50.896919 13199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:50.897249 13191 server_base.cc:1034] running on GCE node
I20260619 02:34:50.897958 13191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:50.899153 13191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:50.900353 13191 hybrid_clock.cc:648] HybridClock initialized: now 1781836490900344 us; error 33 us; skew 500 ppm
I20260619 02:34:50.902514 13191 webserver.cc:460] Webserver started at http://127.7.194.254:37797/ using document root <none> and password file <none>
I20260619 02:34:50.903144 13191 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:50.903237 13191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:50.903478 13191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:50.905730 13191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "39097923b7c84e7da75c3ef72c58c257"
format_stamp: "Formatted at 2026-06-19 02:34:50 on dist-test-slave-9tkz"
server_key: "74f87a5ea12e2f7c681314338e510190"
server_key_iv: "5c3df04888371fb030db8815534dbd74"
server_key_version: "encryptionkey@0"
I20260619 02:34:50.906327 13191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "39097923b7c84e7da75c3ef72c58c257"
format_stamp: "Formatted at 2026-06-19 02:34:50 on dist-test-slave-9tkz"
server_key: "74f87a5ea12e2f7c681314338e510190"
server_key_iv: "5c3df04888371fb030db8815534dbd74"
server_key_version: "encryptionkey@0"
I20260619 02:34:50.910295 13191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:50.912894 13205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:50.914309 13191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:50.914461 13191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "39097923b7c84e7da75c3ef72c58c257"
format_stamp: "Formatted at 2026-06-19 02:34:50 on dist-test-slave-9tkz"
server_key: "74f87a5ea12e2f7c681314338e510190"
server_key_iv: "5c3df04888371fb030db8815534dbd74"
server_key_version: "encryptionkey@0"
I20260619 02:34:50.914587 13191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:50.934041 13191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:50.934849 13191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:50.935065 13191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:50.942127 13191 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:40535
I20260619 02:34:50.942139 13257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:40535 every 8 connection(s)
I20260619 02:34:50.943166 13191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:50.946372 13258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:50.947474 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 13191
I20260619 02:34:50.947611 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:50.947976 7947 external_mini_cluster.cc:1428] Setting key 5ed250748b04055642393e19a47b2bba
I20260619 02:34:50.953014 13258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257: Bootstrap starting.
I20260619 02:34:50.955896 13258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:50.956704 13258 log.cc:824] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:50.958165 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:50.949515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.7.194.254:40535)
0619 02:34:50.950200 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:50.950230 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:50.950268 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:34:50.950921 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:50.950938 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:50.951357 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:50.951643 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:50.952117 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:50.953067 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:50.954448 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:50.955025 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:50.955185 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:50.956139 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:50.956144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:50.956157 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:50.956595 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:50.956652 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:50.956672 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:50.956758 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:50.957087 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 02:34:50.957369 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":145,"threads_started":1}
I20260619 02:34:50.958983 13258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257: No bootstrap required, opened a new log
I20260619 02:34:50.961025 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:34:50.963092 13258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39097923b7c84e7da75c3ef72c58c257" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40535 } }
I20260619 02:34:50.963330 13258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:50.963408 13258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39097923b7c84e7da75c3ef72c58c257, State: Initialized, Role: FOLLOWER
I20260619 02:34:50.963867 13258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39097923b7c84e7da75c3ef72c58c257" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40535 } }
I20260619 02:34:50.964012 13258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:50.964078 13258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:50.964196 13258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:50.965168 13258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39097923b7c84e7da75c3ef72c58c257" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40535 } }
I20260619 02:34:50.965519 13258 leader_election.cc:302] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39097923b7c84e7da75c3ef72c58c257; no voters:
I20260619 02:34:50.965940 13258 leader_election.cc:288] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:50.966064 13263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:50.966320 13263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [term 1 LEADER]: Becoming Leader. State: Replica: 39097923b7c84e7da75c3ef72c58c257, State: Running, Role: LEADER
I20260619 02:34:50.966639 13263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39097923b7c84e7da75c3ef72c58c257" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40535 } }
I20260619 02:34:50.967355 13258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:50.968384 13264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39097923b7c84e7da75c3ef72c58c257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39097923b7c84e7da75c3ef72c58c257" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40535 } } }
I20260619 02:34:50.968396 13265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39097923b7c84e7da75c3ef72c58c257. Latest consensus state: current_term: 1 leader_uuid: "39097923b7c84e7da75c3ef72c58c257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39097923b7c84e7da75c3ef72c58c257" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40535 } } }
I20260619 02:34:50.968540 13264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:50.968545 13265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:50.968894 13272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:50.972255 13272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:50.978078 13272 catalog_manager.cc:1347] Generated new cluster ID: c21e9df8492b429e8db3bcbaa5999211
I20260619 02:34:50.978158 13272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:51.000312 13272 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:51.001816 13272 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:51.013659 13272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39097923b7c84e7da75c3ef72c58c257: Generated new TSK 0
I20260619 02:34:51.014596 13272 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:34:51.050467 13288 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 02:34:51.050580 13288 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:34:51.051364 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:51.034934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.7.194.254:40535)
0619 02:34:51.035123 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:51.035128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:51.035226 (+ 98us) server_negotiation.cc:407] Connection header received
0619 02:34:51.035398 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:51.035401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:51.035466 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:51.035595 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:34:51.035816 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.036694 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.037710 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.038074 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.038200 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:51.038500 (+ 300us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:34:51.050904 (+ 12404us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:34:51.051201 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48186: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 02:34:51.051436 13261 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48186: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 02:34:51.051453 13287 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:34:51.034826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40535 (local address 127.0.0.1:48186)
0619 02:34:51.035132 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:51.035145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:51.035258 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:51.035605 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:51.035607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:51.035613 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:34:51.035715 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:51.035721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.036849 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.036853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:51.037529 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:51.037537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.038213 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.038216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:51.038291 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:51.038316 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:34:51.051122 (+ 12806us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:34:51.051287 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.194.254:40535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":119,"threads_started":1}
I20260619 02:34:51.052296 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 13191
2026-06-19T02:34:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 02:34:51.091876 7947 mini_oidc.cc:152] Starting JWKS server
I20260619 02:34:51.092732 7947 webserver.cc:460] Webserver started at https://127.0.0.1:36407/ using document root <none> and password file <none>
I20260619 02:34:51.092815 7947 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:34:51.093075 7947 webserver.cc:460] Webserver started at http://127.0.0.1:33311/ using document root <none> and password file <none>
2026-06-19T02:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:51Z Disabled control of system clock
I20260619 02:34:51.101722 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:41609
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:41609
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36407/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:34:51.216784 13294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:51.217115 13294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:51.217193 13294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:51.221340 13294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:51.221449 13294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:51.221482 13294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:51.221522 13294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:51.221573 13294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:34:51.221643 13294 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:34:51.221676 13294 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36407/jwks.json
I20260619 02:34:51.226662 13294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:41609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36407/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:51.227854 13294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:51.229367 13294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:51.237963 13299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:51.237978 13302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:51.237977 13300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:51.238373 13294 server_base.cc:1034] running on GCE node
I20260619 02:34:51.239045 13294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:51.240247 13294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:51.241448 13294 hybrid_clock.cc:648] HybridClock initialized: now 1781836491241429 us; error 48 us; skew 500 ppm
I20260619 02:34:51.246438 13294 webserver.cc:460] Webserver started at http://127.7.194.254:34533/ using document root <none> and password file <none>
I20260619 02:34:51.247092 13294 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:51.247187 13294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:51.247427 13294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:51.249683 13294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "1376492d5f464aa5a28e844f625eab91"
format_stamp: "Formatted at 2026-06-19 02:34:51 on dist-test-slave-9tkz"
server_key: "e099b277f801394e6f48a42cdd0d3e11"
server_key_iv: "614ee1717f3b73f7352d22285b6792e6"
server_key_version: "encryptionkey@0"
I20260619 02:34:51.250283 13294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "1376492d5f464aa5a28e844f625eab91"
format_stamp: "Formatted at 2026-06-19 02:34:51 on dist-test-slave-9tkz"
server_key: "e099b277f801394e6f48a42cdd0d3e11"
server_key_iv: "614ee1717f3b73f7352d22285b6792e6"
server_key_version: "encryptionkey@0"
I20260619 02:34:51.254171 13294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:34:51.256698 13308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:51.258225 13294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:51.258371 13294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "1376492d5f464aa5a28e844f625eab91"
format_stamp: "Formatted at 2026-06-19 02:34:51 on dist-test-slave-9tkz"
server_key: "e099b277f801394e6f48a42cdd0d3e11"
server_key_iv: "614ee1717f3b73f7352d22285b6792e6"
server_key_version: "encryptionkey@0"
I20260619 02:34:51.258488 13294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:51.287098 13294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:51.287910 13294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:51.288136 13294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:51.295593 13294 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:41609
I20260619 02:34:51.295670 13360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:41609 every 8 connection(s)
I20260619 02:34:51.296670 13294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:51.297899 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 13294
I20260619 02:34:51.298045 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:51.298349 7947 external_mini_cluster.cc:1428] Setting key cab3985dd22b136445628e06f727143b
I20260619 02:34:51.300442 13361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:51.307672 13361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91: Bootstrap starting.
I20260619 02:34:51.309103 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:51.299779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.7.194.254:41609)
0619 02:34:51.300489 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:51.300509 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:51.300555 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:34:51.301435 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:51.301459 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:51.301992 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:51.302382 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:51.302829 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.304326 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.305225 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.305961 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.306133 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:51.307118 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:51.307127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:51.307149 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:51.307728 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:51.307829 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:51.307859 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:51.307973 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:51.308090 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 02:34:51.308373 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
I20260619 02:34:51.310242 13361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:51.311165 13361 log.cc:824] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:51.312153 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:34:51.313478 13361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91: No bootstrap required, opened a new log
I20260619 02:34:51.316975 13361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376492d5f464aa5a28e844f625eab91" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41609 } }
I20260619 02:34:51.317260 13361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:51.317301 13361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1376492d5f464aa5a28e844f625eab91, State: Initialized, Role: FOLLOWER
I20260619 02:34:51.317739 13361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376492d5f464aa5a28e844f625eab91" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41609 } }
I20260619 02:34:51.317858 13361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:51.317906 13361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:51.317977 13361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:51.319173 13361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376492d5f464aa5a28e844f625eab91" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41609 } }
I20260619 02:34:51.319471 13361 leader_election.cc:302] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1376492d5f464aa5a28e844f625eab91; no voters:
I20260619 02:34:51.319736 13361 leader_election.cc:288] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:51.319887 13366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:51.320118 13366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [term 1 LEADER]: Becoming Leader. State: Replica: 1376492d5f464aa5a28e844f625eab91, State: Running, Role: LEADER
I20260619 02:34:51.320456 13366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376492d5f464aa5a28e844f625eab91" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41609 } }
I20260619 02:34:51.321067 13361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:51.322228 13368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1376492d5f464aa5a28e844f625eab91. Latest consensus state: current_term: 1 leader_uuid: "1376492d5f464aa5a28e844f625eab91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376492d5f464aa5a28e844f625eab91" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41609 } } }
I20260619 02:34:51.322454 13368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:51.322918 13367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1376492d5f464aa5a28e844f625eab91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376492d5f464aa5a28e844f625eab91" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41609 } } }
I20260619 02:34:51.323016 13367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:51.323444 13375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:51.326159 13375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:51.332337 13375 catalog_manager.cc:1347] Generated new cluster ID: 7357ddf644ad4b76be9cbbe4b97a9194
I20260619 02:34:51.332432 13375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:51.348258 13375 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:51.349665 13375 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:51.360730 13375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1376492d5f464aa5a28e844f625eab91: Generated new TSK 0
I20260619 02:34:51.361670 13375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:34:51.376917 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:51.371727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.7.194.254:41609)
0619 02:34:51.371880 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:51.371887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:51.372079 (+ 192us) server_negotiation.cc:407] Connection header received
0619 02:34:51.372288 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:51.372291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:51.372340 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:51.372448 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:34:51.372715 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.373243 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.374509 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.374755 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.374839 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:51.375178 (+ 339us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:34:51.376508 (+ 1330us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:34:51.376773 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 02:34:51.377008 13364 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 02:34:51.379815 13390 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:34:51.371626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:41609 (local address 127.0.0.1:55504)
0619 02:34:51.371979 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:51.371995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:51.372115 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:51.372471 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:51.372475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:51.372483 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:34:51.372607 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:51.372613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.373361 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.373364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:51.374378 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:51.374389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.374886 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.374890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:51.374980 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:51.375009 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:34:51.376695 (+ 1686us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:34:51.379656 (+ 2961us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.194.254:41609: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":2737,"thread_start_us":110,"threads_started":1}
I20260619 02:34:51.380537 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 13294
E20260619 02:34:51.388996 13391 webserver.cc:533] Webserver: sslize failed:
E20260619 02:34:51.389092 13391 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T02:34:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T02:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:51Z Disabled control of system clock
I20260619 02:34:51.428401 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:43813
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:43813
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/testchainca.cert with env {}
W20260619 02:34:51.547492 13395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:51.547941 13395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:51.548007 13395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:51.552196 13395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:51.552276 13395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:51.552292 13395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:51.552309 13395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:51.552325 13395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:34:51.552341 13395 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:34:51.552402 13395 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 02:34:51.558233 13395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:43813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:43813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:51.559543 13395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:51.560662 13395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:51.569427 13400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:51.569478 13403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:51.569546 13395 server_base.cc:1034] running on GCE node
W20260619 02:34:51.569446 13401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:51.570390 13395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:51.571537 13395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:51.572711 13395 hybrid_clock.cc:648] HybridClock initialized: now 1781836491572676 us; error 47 us; skew 500 ppm
I20260619 02:34:51.574868 13395 webserver.cc:460] Webserver started at http://127.7.194.254:38325/ using document root <none> and password file <none>
I20260619 02:34:51.575420 13395 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:51.575479 13395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:51.575647 13395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:51.577785 13395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "d3d2149d4a654dd68afa522a09640eff"
format_stamp: "Formatted at 2026-06-19 02:34:51 on dist-test-slave-9tkz"
server_key: "4379cfe8b1598e1d0e6eb8bb94af1b1f"
server_key_iv: "c799e50bc68d262832810d8a3edfdb8b"
server_key_version: "encryptionkey@0"
I20260619 02:34:51.578343 13395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "d3d2149d4a654dd68afa522a09640eff"
format_stamp: "Formatted at 2026-06-19 02:34:51 on dist-test-slave-9tkz"
server_key: "4379cfe8b1598e1d0e6eb8bb94af1b1f"
server_key_iv: "c799e50bc68d262832810d8a3edfdb8b"
server_key_version: "encryptionkey@0"
I20260619 02:34:51.582181 13395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:34:51.584766 13409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:51.586298 13395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:51.586436 13395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "d3d2149d4a654dd68afa522a09640eff"
format_stamp: "Formatted at 2026-06-19 02:34:51 on dist-test-slave-9tkz"
server_key: "4379cfe8b1598e1d0e6eb8bb94af1b1f"
server_key_iv: "c799e50bc68d262832810d8a3edfdb8b"
server_key_version: "encryptionkey@0"
I20260619 02:34:51.586550 13395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:51.611498 13395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:51.612288 13395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:51.612500 13395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:51.620193 13395 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:43813
I20260619 02:34:51.620213 13461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:43813 every 8 connection(s)
I20260619 02:34:51.621222 13395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:51.624511 13462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:51.625166 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 13395
I20260619 02:34:51.625299 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:51.625631 7947 external_mini_cluster.cc:1428] Setting key 6953e5c29b73a43724449291be853135
I20260619 02:34:51.631145 13462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff: Bootstrap starting.
I20260619 02:34:51.633862 13462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:51.634707 13462 log.cc:824] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:51.635442 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:51.626971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.7.194.254:43813)
0619 02:34:51.627605 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:51.627619 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:51.627659 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:34:51.628318 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:51.628336 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:51.628760 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:51.629067 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:51.629422 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.630478 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.631632 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.632112 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.632240 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:51.633216 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:51.633221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:51.633236 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:34:51.633724 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:51.633795 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:34:51.633830 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:51.633913 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:51.634148 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 02:34:51.634462 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":137,"threads_started":1}
I20260619 02:34:51.637009 13462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff: No bootstrap required, opened a new log
I20260619 02:34:51.638500 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:34:51.641142 13462 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2149d4a654dd68afa522a09640eff" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43813 } }
I20260619 02:34:51.641420 13462 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:51.641502 13462 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d2149d4a654dd68afa522a09640eff, State: Initialized, Role: FOLLOWER
I20260619 02:34:51.641968 13462 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2149d4a654dd68afa522a09640eff" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43813 } }
I20260619 02:34:51.642123 13462 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:51.642217 13462 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:51.642338 13462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:51.643333 13462 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2149d4a654dd68afa522a09640eff" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43813 } }
I20260619 02:34:51.643689 13462 leader_election.cc:302] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d2149d4a654dd68afa522a09640eff; no voters:
I20260619 02:34:51.644021 13462 leader_election.cc:288] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:51.644122 13467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:51.644399 13467 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [term 1 LEADER]: Becoming Leader. State: Replica: d3d2149d4a654dd68afa522a09640eff, State: Running, Role: LEADER
I20260619 02:34:51.644733 13467 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2149d4a654dd68afa522a09640eff" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43813 } }
I20260619 02:34:51.645596 13462 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:51.646608 13469 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d2149d4a654dd68afa522a09640eff. Latest consensus state: current_term: 1 leader_uuid: "d3d2149d4a654dd68afa522a09640eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2149d4a654dd68afa522a09640eff" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43813 } } }
I20260619 02:34:51.646656 13468 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3d2149d4a654dd68afa522a09640eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2149d4a654dd68afa522a09640eff" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43813 } } }
I20260619 02:34:51.646782 13468 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:51.646782 13469 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:51.647269 13476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:51.650159 13476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:51.655957 13476 catalog_manager.cc:1347] Generated new cluster ID: d4d0054a5b364cae9551038a33d389af
I20260619 02:34:51.656038 13476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:51.668768 13476 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:51.670221 13476 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:51.680204 13476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3d2149d4a654dd68afa522a09640eff: Generated new TSK 0
I20260619 02:34:51.681143 13476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:34:51.771711 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:51.690118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.7.194.254:43813)
0619 02:34:51.690259 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:51.690266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:51.690565 (+ 299us) server_negotiation.cc:407] Connection header received
0619 02:34:51.690730 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:51.690734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:51.690801 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:51.690932 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:34:51.691205 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.692030 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.692968 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.693329 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.693447 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:51.693957 (+ 510us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:34:51.771130 (+ 77173us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:34:51.771526 (+ 396us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45172: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 02:34:51.771767 13491 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:34:51.690047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:43813 (local address 127.0.0.1:45172)
0619 02:34:51.690472 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:51.690484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:51.690582 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:51.690953 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:51.690956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:51.690964 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:34:51.691090 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:51.691098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.692175 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.692178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:51.692850 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:51.692859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:51.693473 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:51.693477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:51.693614 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:51.693657 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:34:51.771408 (+ 77751us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:34:51.771606 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.194.254:43813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":125,"threads_started":1}
W20260619 02:34:51.771829 13465 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45172: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260619 02:34:51.772647 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 13395
2026-06-19T02:34:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (372 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:52 dist-test-slave-9tkz krb5kdc[13503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:52 dist-test-slave-9tkz krb5kdc[13503](info): set up 2 sockets
Jun 19 02:34:52 dist-test-slave-9tkz krb5kdc[13503](info): commencing operation
krb5kdc: starting...
W20260619 02:34:53.069973 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:53 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:34:53.196440 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:34181
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:34681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:34181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:53.315054 13516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:53.315356 13516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:53.315398 13516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:53.319646 13516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:53.319717 13516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:53.319737 13516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:53.319754 13516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:53.319769 13516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:53.325280 13516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:34681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:34181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:34181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:53.326524 13516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:53.327612 13516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:53.336395 13524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:53.336517 13516 server_base.cc:1034] running on GCE node
W20260619 02:34:53.336474 13522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:53.336395 13521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:53.337306 13516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:53.338507 13516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:53.339715 13516 hybrid_clock.cc:648] HybridClock initialized: now 1781836493339696 us; error 33 us; skew 500 ppm
Jun 19 02:34:53 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836493, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:53.342669 13516 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:34:53.343555 13516 webserver.cc:460] Webserver started at http://127.7.194.254:33211/ using document root <none> and password file <none>
I20260619 02:34:53.344095 13516 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:53.344159 13516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:53.344332 13516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:53.346517 13516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "45e75ba624dd4df886f7c49201f7c0c8"
format_stamp: "Formatted at 2026-06-19 02:34:53 on dist-test-slave-9tkz"
server_key: "7d99a38903d5463558db2e63f00bd2b5"
server_key_iv: "88d0d509ff234ad72a0b764235b1f839"
server_key_version: "encryptionkey@0"
I20260619 02:34:53.347076 13516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "45e75ba624dd4df886f7c49201f7c0c8"
format_stamp: "Formatted at 2026-06-19 02:34:53 on dist-test-slave-9tkz"
server_key: "7d99a38903d5463558db2e63f00bd2b5"
server_key_iv: "88d0d509ff234ad72a0b764235b1f839"
server_key_version: "encryptionkey@0"
I20260619 02:34:53.351217 13516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:53.353822 13531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:53.355259 13516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:53.355500 13516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "45e75ba624dd4df886f7c49201f7c0c8"
format_stamp: "Formatted at 2026-06-19 02:34:53 on dist-test-slave-9tkz"
server_key: "7d99a38903d5463558db2e63f00bd2b5"
server_key_iv: "88d0d509ff234ad72a0b764235b1f839"
server_key_version: "encryptionkey@0"
I20260619 02:34:53.355621 13516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:53.377141 13516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:53.380602 13516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:53.380828 13516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:53.388326 13516 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:34181
I20260619 02:34:53.388350 13583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:34181 every 8 connection(s)
I20260619 02:34:53.389349 13516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:53.392426 13584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:53.393448 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 13516
I20260619 02:34:53.393620 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:53.393925 7947 external_mini_cluster.cc:1428] Setting key 57b389a329ff6c1f72f10449da21f89f
I20260619 02:34:53.399148 13584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8: Bootstrap starting.
I20260619 02:34:53.402158 13584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:53.403146 13584 log.cc:824] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:34:53 dist-test-slave-9tkz krb5kdc[13503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:53.405390 13584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8: No bootstrap required, opened a new log
I20260619 02:34:53.409034 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:53.395483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.7.194.254:34181)
0619 02:34:53.395939 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:53.395954 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:53.396001 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:34:53.396665 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:53.396684 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:53.397034 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:53.397352 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:53.398332 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.399361 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.400516 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.401011 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.401141 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:53.404138 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:53.404147 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:53.404168 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:53.404196 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:53.406118 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:53.406631 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:53.406637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:53.406650 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:53.406761 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:53.407045 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:53.407048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:53.407050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:53.407456 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:53.407659 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:53.407958 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 02:34:53.408278 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":135,"threads_started":1}
I20260619 02:34:53.409137 13584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e75ba624dd4df886f7c49201f7c0c8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34181 } }
I20260619 02:34:53.409431 13584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:53.409516 13584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e75ba624dd4df886f7c49201f7c0c8, State: Initialized, Role: FOLLOWER
I20260619 02:34:53.410041 13584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e75ba624dd4df886f7c49201f7c0c8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34181 } }
I20260619 02:34:53.410207 13584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:53.410279 13584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:53.410413 13584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:53.411407 13584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e75ba624dd4df886f7c49201f7c0c8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34181 } }
I20260619 02:34:53.411746 13584 leader_election.cc:302] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45e75ba624dd4df886f7c49201f7c0c8; no voters:
I20260619 02:34:53.412052 13584 leader_election.cc:288] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:53.412186 13589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:53.412436 13589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [term 1 LEADER]: Becoming Leader. State: Replica: 45e75ba624dd4df886f7c49201f7c0c8, State: Running, Role: LEADER
I20260619 02:34:53.412770 13589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e75ba624dd4df886f7c49201f7c0c8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34181 } }
I20260619 02:34:53.413305 13584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:53.414454 13590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45e75ba624dd4df886f7c49201f7c0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e75ba624dd4df886f7c49201f7c0c8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34181 } } }
I20260619 02:34:53.414592 13590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:53.415002 13598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:53.414945 13591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e75ba624dd4df886f7c49201f7c0c8. Latest consensus state: current_term: 1 leader_uuid: "45e75ba624dd4df886f7c49201f7c0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e75ba624dd4df886f7c49201f7c0c8" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34181 } } }
I20260619 02:34:53.415177 13591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:53.418272 13598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:53.423730 13598 catalog_manager.cc:1347] Generated new cluster ID: 2a8a2bf5407b499eb694898f4ba77f43
I20260619 02:34:53.423804 13598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:53.444337 13598 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:53.445813 13598 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:53.454444 13598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45e75ba624dd4df886f7c49201f7c0c8: Generated new TSK 0
I20260619 02:34:53.455412 13598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:34:53.490561 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34181
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:34681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:53.608858 13610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:53.609153 13610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:53.609198 13610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:53.613413 13610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:53.613528 13610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:53.613687 13610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:53.619297 13610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:34681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:53.620474 13610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:53.621593 13610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:53.631772 13618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:53.631795 13610 server_base.cc:1034] running on GCE node
W20260619 02:34:53.631855 13615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:53.631855 13616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:53.632567 13610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:53.633257 13610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:53.634436 13610 hybrid_clock.cc:648] HybridClock initialized: now 1781836493634396 us; error 53 us; skew 500 ppm
Jun 19 02:34:53 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836493, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:53.637537 13610 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:34:53.638600 13610 webserver.cc:460] Webserver started at http://127.7.194.193:35779/ using document root <none> and password file <none>
I20260619 02:34:53.639216 13610 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:53.639317 13610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:53.639560 13610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:53.641799 13610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "0d41ace4a27947d1ae9861c30467394c"
format_stamp: "Formatted at 2026-06-19 02:34:53 on dist-test-slave-9tkz"
server_key: "3d86ef2a6523c8b4bdf38f061df424bb"
server_key_iv: "b48af3949556ee14ec5fc30f83e87d50"
server_key_version: "encryptionkey@0"
I20260619 02:34:53.642421 13610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "0d41ace4a27947d1ae9861c30467394c"
format_stamp: "Formatted at 2026-06-19 02:34:53 on dist-test-slave-9tkz"
server_key: "3d86ef2a6523c8b4bdf38f061df424bb"
server_key_iv: "b48af3949556ee14ec5fc30f83e87d50"
server_key_version: "encryptionkey@0"
I20260619 02:34:53.646347 13610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:34:53.648870 13625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:53.650195 13610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 02:34:53.650358 13610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "0d41ace4a27947d1ae9861c30467394c"
format_stamp: "Formatted at 2026-06-19 02:34:53 on dist-test-slave-9tkz"
server_key: "3d86ef2a6523c8b4bdf38f061df424bb"
server_key_iv: "b48af3949556ee14ec5fc30f83e87d50"
server_key_version: "encryptionkey@0"
I20260619 02:34:53.650477 13610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:53.675911 13610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:53.679544 13610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:53.679788 13610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:53.680424 13610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:53.681327 13610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:53.681401 13610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:53.681476 13610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:53.681528 13610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:53.690276 13610 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:32929
I20260619 02:34:53.690305 13738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:32929 every 8 connection(s)
I20260619 02:34:53.691234 13610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:53.698617 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 13610
I20260619 02:34:53.698755 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:53.699066 7947 external_mini_cluster.cc:1428] Setting key 17acc5004f09e29e97d9a52c37de0e91
Jun 19 02:34:53 dist-test-slave-9tkz krb5kdc[13503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836493, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:53.705739 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:53.693087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:52115 (local address 127.7.194.254:34181)
0619 02:34:53.693237 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:53.693242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:53.694133 (+ 891us) server_negotiation.cc:407] Connection header received
0619 02:34:53.695573 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:53.695577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:53.695634 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:53.695723 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:53.696941 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.697479 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.698926 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.699214 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.699305 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:53.702033 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:53.702036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:53.702038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:53.702060 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:53.703586 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:53.704120 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:53.704125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:53.704127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:53.704217 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:53.704578 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:53.704582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:53.704584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:53.704779 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:53.704897 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:53.705412 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 02:34:53.705566 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:34:53.706703 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:53.693509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:34181 (local address 127.7.194.193:52115)
0619 02:34:53.693979 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:53.694023 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:53.695336 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:53.695918 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:53.695926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:53.696302 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:53.696790 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:53.696804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.697654 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.697662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:53.698780 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:53.698792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.699373 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.699378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:53.699589 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:53.700219 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:53.700242 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:53.701857 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:53.703718 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:53.703724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:53.703742 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:53.704014 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:53.704318 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:53.704321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:53.704323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:53.704476 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:53.704893 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:53.704901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:53.705171 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 02:34:53.705372 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 02:34:53.705772 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":112,"threads_started":1}
I20260619 02:34:53.708040 13739 heartbeater.cc:347] Connected to a master server at 127.7.194.254:34181
I20260619 02:34:53.708318 13739 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:34:53.708952 13739 heartbeater.cc:511] Master 127.7.194.254:34181 requested a full tablet report, sending...
I20260619 02:34:53.710634 13548 ts_manager.cc:194] Registered new tserver with Master: 0d41ace4a27947d1ae9861c30467394c (127.7.194.193:32929)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:34:53.711872 13548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:52115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:34:53.728313 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34181
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:34681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:53.846181 13744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:53.846464 13744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:53.846509 13744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:53.850742 13744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:53.850813 13744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:53.850876 13744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:53.856422 13744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:34681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:53.857529 13744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:53.858668 13744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:53.868721 13750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:53.868969 13744 server_base.cc:1034] running on GCE node
W20260619 02:34:53.868726 13752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:53.868757 13749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:53.869505 13744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:53.870281 13744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:53.871459 13744 hybrid_clock.cc:648] HybridClock initialized: now 1781836493871440 us; error 32 us; skew 500 ppm
Jun 19 02:34:53 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836493, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:53.874758 13744 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:34:53.875706 13744 webserver.cc:460] Webserver started at http://127.7.194.194:38043/ using document root <none> and password file <none>
I20260619 02:34:53.876292 13744 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:53.876355 13744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:53.876574 13744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:53.878813 13744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "a01cd32ecd7442cc844d9788cfc97a0f"
format_stamp: "Formatted at 2026-06-19 02:34:53 on dist-test-slave-9tkz"
server_key: "8b01282d7827b6bb861cea528b597219"
server_key_iv: "e577d573f5370c1fb8e650a4ec907cd4"
server_key_version: "encryptionkey@0"
I20260619 02:34:53.879379 13744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "a01cd32ecd7442cc844d9788cfc97a0f"
format_stamp: "Formatted at 2026-06-19 02:34:53 on dist-test-slave-9tkz"
server_key: "8b01282d7827b6bb861cea528b597219"
server_key_iv: "e577d573f5370c1fb8e650a4ec907cd4"
server_key_version: "encryptionkey@0"
I20260619 02:34:53.883352 13744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:53.886096 13759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:53.887537 13744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:53.887698 13744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "a01cd32ecd7442cc844d9788cfc97a0f"
format_stamp: "Formatted at 2026-06-19 02:34:53 on dist-test-slave-9tkz"
server_key: "8b01282d7827b6bb861cea528b597219"
server_key_iv: "e577d573f5370c1fb8e650a4ec907cd4"
server_key_version: "encryptionkey@0"
I20260619 02:34:53.887861 13744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:53.916159 13744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:53.919564 13744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:53.919803 13744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:53.920411 13744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:53.921271 13744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:53.921316 13744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:53.921384 13744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:53.921419 13744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:53.930644 13744 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:38087
I20260619 02:34:53.930662 13872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:38087 every 8 connection(s)
I20260619 02:34:53.931614 13744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:53.935359 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 13744
I20260619 02:34:53.935523 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:53.935856 7947 external_mini_cluster.cc:1428] Setting key a12b0207520d9c91ac36c078a1735833
Jun 19 02:34:53 dist-test-slave-9tkz krb5kdc[13503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836493, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:34:53.946398 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:53.933496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:33587 (local address 127.7.194.254:34181)
0619 02:34:53.933646 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:53.933650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:53.934667 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 02:34:53.936248 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:53.936252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:53.936313 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:53.936425 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:53.937734 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.938260 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.939463 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.939690 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.939803 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:53.942373 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:53.942376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:53.942378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:53.942402 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:53.944007 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:53.944555 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:53.944558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:53.944560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:53.944638 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:53.945040 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:53.945044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:53.945046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:53.945318 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:53.945406 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:53.946144 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 02:34:53.946254 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:34:53.947474 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:53.933990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:34181 (local address 127.7.194.194:33587)
0619 02:34:53.934489 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:53.934534 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:53.935988 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:53.936623 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:53.936631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:53.937018 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:53.937522 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:53.937597 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.938391 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.938395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:53.939334 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:53.939343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:53.939843 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:53.939847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:53.940018 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:53.940652 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:53.940669 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:53.942180 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:53.944137 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:53.944144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:53.944161 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:53.944427 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:53.944740 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:53.944744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:53.944746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:53.944894 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:53.945440 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:53.945446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:53.945822 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 02:34:53.946063 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 02:34:53.946359 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":107,"threads_started":1}
I20260619 02:34:53.948781 13873 heartbeater.cc:347] Connected to a master server at 127.7.194.254:34181
I20260619 02:34:53.949102 13873 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:53.949800 13873 heartbeater.cc:511] Master 127.7.194.254:34181 requested a full tablet report, sending...
I20260619 02:34:53.950977 13548 ts_manager.cc:194] Registered new tserver with Master: a01cd32ecd7442cc844d9788cfc97a0f (127.7.194.194:38087)
I20260619 02:34:53.951581 13548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:33587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:34:53.964777 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34181
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:34681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:54.080935 13878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:54.081228 13878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:54.081274 13878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:54.085486 13878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:54.085618 13878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:54.085728 13878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:54.091492 13878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:34681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:54.092721 13878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:54.093900 13878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:54.103902 13884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:54.103904 13886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:54.103902 13883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:54.104099 13878 server_base.cc:1034] running on GCE node
I20260619 02:34:54.104679 13878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:54.105340 13878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:54.106506 13878 hybrid_clock.cc:648] HybridClock initialized: now 1781836494106476 us; error 43 us; skew 500 ppm
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:54.109347 13878 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:34:54.110375 13878 webserver.cc:460] Webserver started at http://127.7.194.195:33941/ using document root <none> and password file <none>
I20260619 02:34:54.111027 13878 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:54.111119 13878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:54.111361 13878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:54.113663 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "91568c9330c14250a9da698269ed7e4d"
format_stamp: "Formatted at 2026-06-19 02:34:54 on dist-test-slave-9tkz"
server_key: "ab2275478fad27ddb5e3f8987c5d72d0"
server_key_iv: "50d20b5747bbe99a65620942de58752f"
server_key_version: "encryptionkey@0"
I20260619 02:34:54.114324 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "91568c9330c14250a9da698269ed7e4d"
format_stamp: "Formatted at 2026-06-19 02:34:54 on dist-test-slave-9tkz"
server_key: "ab2275478fad27ddb5e3f8987c5d72d0"
server_key_iv: "50d20b5747bbe99a65620942de58752f"
server_key_version: "encryptionkey@0"
I20260619 02:34:54.118396 13878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:34:54.121002 13893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:54.122442 13878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:34:54.122598 13878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "91568c9330c14250a9da698269ed7e4d"
format_stamp: "Formatted at 2026-06-19 02:34:54 on dist-test-slave-9tkz"
server_key: "ab2275478fad27ddb5e3f8987c5d72d0"
server_key_iv: "50d20b5747bbe99a65620942de58752f"
server_key_version: "encryptionkey@0"
I20260619 02:34:54.122735 13878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:54.133644 13878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:54.136703 13878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:54.136906 13878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:54.137518 13878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:54.138449 13878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:54.138520 13878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:54.138599 13878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:54.138645 13878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:54.147528 13878 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:44797
I20260619 02:34:54.147621 14006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:44797 every 8 connection(s)
I20260619 02:34:54.148561 13878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:54.151569 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 13878
I20260619 02:34:54.151721 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:54.152071 7947 external_mini_cluster.cc:1428] Setting key 81085f6da5870df79fc9d2b2567758fa
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:34:54.163589 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:54.150554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:44473 (local address 127.7.194.254:34181)
0619 02:34:54.150727 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:54.150740 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:54.151755 (+ 1015us) server_negotiation.cc:407] Connection header received
0619 02:34:54.153580 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:54.153584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:54.153647 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:54.153753 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:54.155121 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.155720 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.156901 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.157130 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.157250 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:54.159886 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:54.159890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:54.159894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:54.159925 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:54.161444 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:54.162017 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:54.162022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:54.162024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:54.162108 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:54.162466 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:54.162469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:54.162471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:54.162662 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:54.162767 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:54.163290 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 02:34:54.163419 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:34:54.164564 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:54.150962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:34181 (local address 127.7.194.195:44473)
0619 02:34:54.151526 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:54.151590 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:54.153304 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:54.153947 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:54.153954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:54.154431 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:54.154973 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:54.154988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.155844 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.155848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:54.156762 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:54.156771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.157287 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.157291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:54.157467 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:54.158159 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:54.158178 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:54.159715 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:54.161596 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:54.161604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:54.161623 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:54.161909 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:54.162209 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:54.162213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:54.162216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:54.162362 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:54.162777 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:54.162786 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:54.163048 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 02:34:54.163250 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 02:34:54.163598 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":116,"threads_started":1}
I20260619 02:34:54.166443 14007 heartbeater.cc:347] Connected to a master server at 127.7.194.254:34181
I20260619 02:34:54.166761 14007 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:54.167315 14007 heartbeater.cc:511] Master 127.7.194.254:34181 requested a full tablet report, sending...
I20260619 02:34:54.168419 13548 ts_manager.cc:194] Registered new tserver with Master: 91568c9330c14250a9da698269ed7e4d (127.7.194.195:44797)
I20260619 02:34:54.168970 13548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:44473
I20260619 02:34:54.179057 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/test-admin.keytab.
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:54.194490 7947 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 02:34:54.197521 14014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:34:54.196031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.7.194.193:32929)
0619 02:34:54.196369 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:54.196385 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:54.196410 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:34:54.196461 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:54.196467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:54.196661 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:54.196795 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:54.197342 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51222: BlockingRecv error: recv got EOF from 127.0.0.1:51222 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":143,"threads_started":1}
W20260619 02:34:54.197976 7947 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:54.199143 7947 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:54.200114 7947 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:54.210348 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:54.201111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.7.194.193:32929)
0619 02:34:54.201260 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:54.201263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:54.201277 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:54.201386 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:54.201389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:54.201430 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:54.201500 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:54.202064 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.202855 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.203652 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.204288 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.204384 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:54.206779 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:54.206788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:54.206802 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:54.206827 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:54.208600 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:54.209025 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:54.209031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:54.209032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:54.209109 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:54.209338 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:54.209340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:54.209341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:54.209640 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:54.209804 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:54.209981 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 02:34:54.210163 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 02:34:54.214434 14014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:34:54.213389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.7.194.193:32929)
0619 02:34:54.213578 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:54.213582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:54.213670 (+ 88us) server_negotiation.cc:407] Connection header received
0619 02:34:54.213768 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:54.213770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:54.213807 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:54.213871 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:54.214307 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51238: BlockingRecv error: recv got EOF from 127.0.0.1:51238 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 02:34:54.214489 7947 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:54.215612 7947 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:54.216565 7947 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:54.225382 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:54.217442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.7.194.193:32929)
0619 02:34:54.217609 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:54.217613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:54.217744 (+ 131us) server_negotiation.cc:407] Connection header received
0619 02:34:54.217888 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:54.217891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:54.217931 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:54.218024 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:54.218527 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.219100 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.219903 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.220154 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.220263 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:54.222496 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:54.222500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:54.222505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:54.222533 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:54.223769 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:54.224185 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:54.224192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:54.224195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:54.224275 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:54.224548 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:54.224552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:54.224554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:54.224731 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:54.224870 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:54.225094 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:34:54.225211 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 02:34:54.228222 14014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:34:54.227118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.7.194.193:32929)
0619 02:34:54.227247 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:54.227252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:54.227377 (+ 125us) server_negotiation.cc:407] Connection header received
0619 02:34:54.227505 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:54.227508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:54.227547 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:54.227645 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:54.228099 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51254: BlockingRecv error: recv got EOF from 127.0.0.1:51254 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 02:34:54.228389 7947 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781836464269074-7947-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:54.229427 7947 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:54.230350 7947 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[13503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:34:54.239210 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:54.231116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.7.194.193:32929)
0619 02:34:54.231237 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:54.231241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:54.231349 (+ 108us) server_negotiation.cc:407] Connection header received
0619 02:34:54.231488 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:54.231491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:54.231562 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:54.231657 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:54.232133 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.232631 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.233435 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:54.233737 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:54.233849 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:54.236027 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:54.236031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:54.236034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:54.236057 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:54.237541 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:54.238020 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:54.238024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:54.238026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:54.238107 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:54.238388 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:54.238393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:54.238395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:54.238560 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:54.238720 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:54.238947 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 02:34:54.239048 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:34:54.240468 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 13610
I20260619 02:34:54.246146 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 13744
I20260619 02:34:54.251618 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 13878
I20260619 02:34:54.256657 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 13516
2026-06-19T02:34:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2230 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[14027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[14027](info): set up 2 sockets
Jun 19 02:34:54 dist-test-slave-9tkz krb5kdc[14027](info): commencing operation
krb5kdc: starting...
W20260619 02:34:55.298017 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:34:55 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:34:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:34:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.254.keytab.
Entry for principal oryx/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.254.keytab.
I20260619 02:34:55.421353 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:45193
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.254.keytab
--principal=oryx/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:45193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:55.536556 14040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:55.536872 14040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:55.536916 14040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:55.540984 14040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:55.541059 14040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:55.541078 14040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:55.541095 14040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:55.541110 14040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:55.546566 14040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:45193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.254.keytab
--principal=oryx/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:45193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:55.547783 14040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:55.548864 14040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:55.557682 14045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:55.557821 14040 server_base.cc:1034] running on GCE node
W20260619 02:34:55.557718 14046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:55.557677 14048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:55.558588 14040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:55.559785 14040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:55.561017 14040 hybrid_clock.cc:648] HybridClock initialized: now 1781836495560951 us; error 50 us; skew 500 ppm
Jun 19 02:34:55 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:55.565289 14040 init.cc:377] Logged in from keytab as oryx/127.7.194.254@KRBTEST.COM (short username oryx)
I20260619 02:34:55.566869 14040 webserver.cc:460] Webserver started at http://127.7.194.254:35455/ using document root <none> and password file <none>
I20260619 02:34:55.567999 14040 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:55.568146 14040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:55.568624 14040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:55.572167 14040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "71ef3db455d5435da039949bfaa1ac86"
format_stamp: "Formatted at 2026-06-19 02:34:55 on dist-test-slave-9tkz"
server_key: "c83ded9dfeaedf5ae91a298e1e8a0bd1"
server_key_iv: "7ca67a340a4f9f657a9827798910f19a"
server_key_version: "encryptionkey@0"
I20260619 02:34:55.573305 14040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "71ef3db455d5435da039949bfaa1ac86"
format_stamp: "Formatted at 2026-06-19 02:34:55 on dist-test-slave-9tkz"
server_key: "c83ded9dfeaedf5ae91a298e1e8a0bd1"
server_key_iv: "7ca67a340a4f9f657a9827798910f19a"
server_key_version: "encryptionkey@0"
I20260619 02:34:55.580040 14040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 02:34:55.583801 14055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:55.585588 14040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:34:55.585786 14040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "71ef3db455d5435da039949bfaa1ac86"
format_stamp: "Formatted at 2026-06-19 02:34:55 on dist-test-slave-9tkz"
server_key: "c83ded9dfeaedf5ae91a298e1e8a0bd1"
server_key_iv: "7ca67a340a4f9f657a9827798910f19a"
server_key_version: "encryptionkey@0"
I20260619 02:34:55.585942 14040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:55.623071 14040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:55.626184 14040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:55.626350 14040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:55.633266 14040 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:45193
I20260619 02:34:55.633618 14107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:45193 every 8 connection(s)
I20260619 02:34:55.634408 14040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:34:55.637403 14108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:55.638460 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 14040
I20260619 02:34:55.638592 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:34:55.638906 7947 external_mini_cluster.cc:1428] Setting key e217c7b7d484f570c33003a434a021fb
I20260619 02:34:55.644956 14108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: Bootstrap starting.
I20260619 02:34:55.647801 14108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:55.648577 14108 log.cc:824] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:34:55 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:34:55.651013 14108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: No bootstrap required, opened a new log
I20260619 02:34:55.654390 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:55.640371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.7.194.254:45193)
0619 02:34:55.640791 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:55.640811 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:55.640860 (+ 49us) server_negotiation.cc:407] Connection header received
0619 02:34:55.641754 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:55.641780 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:55.642218 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:55.642609 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:55.643347 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:55.644832 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:55.645824 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:55.646546 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:55.646710 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:55.649406 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:55.649412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:55.649428 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:55.649458 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:55.651392 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:55.651888 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:55.651894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:55.651898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:55.652016 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:55.652317 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:55.652321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:55.652323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:55.652747 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:55.652955 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:55.653243 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 02:34:55.653619 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":104,"threads_started":1}
I20260619 02:34:55.654796 14108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } }
I20260619 02:34:55.655053 14108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:55.655118 14108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ef3db455d5435da039949bfaa1ac86, State: Initialized, Role: FOLLOWER
I20260619 02:34:55.655603 14108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } }
I20260619 02:34:55.655776 14108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:34:55.655854 14108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:34:55.655977 14108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:55.657047 14108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } }
I20260619 02:34:55.657398 14108 leader_election.cc:302] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71ef3db455d5435da039949bfaa1ac86; no voters:
I20260619 02:34:55.657761 14108 leader_election.cc:288] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:34:55.657884 14113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:55.658164 14113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 1 LEADER]: Becoming Leader. State: Replica: 71ef3db455d5435da039949bfaa1ac86, State: Running, Role: LEADER
I20260619 02:34:55.658466 14113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } }
I20260619 02:34:55.659068 14108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:55.660295 14115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ef3db455d5435da039949bfaa1ac86. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } }
I20260619 02:34:55.660305 14114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } }
I20260619 02:34:55.660458 14114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:55.660457 14115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:55.660907 14122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:55.664240 14122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:55.670549 14122 catalog_manager.cc:1347] Generated new cluster ID: e6c6f9ee024b4211a751114d0a536d6e
I20260619 02:34:55.670640 14122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:55.679035 14122 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:34:55.680754 14122 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:55.698647 14122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: Generated new TSK 0
I20260619 02:34:55.699586 14122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab.
Entry for principal oryx/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab.
I20260619 02:34:55.732779 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:45193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab
--principal=oryx/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:55.850093 14134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:55.850440 14134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:55.850534 14134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:55.854619 14134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:55.854707 14134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:55.854792 14134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:55.860478 14134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab
--principal=oryx/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:45193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:55.861764 14134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:55.862869 14134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:34:55.873365 14134 server_base.cc:1034] running on GCE node
W20260619 02:34:55.873268 14139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:55.873268 14142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:55.873365 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:55.874155 14134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:55.874807 14134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:55.876001 14134 hybrid_clock.cc:648] HybridClock initialized: now 1781836495875975 us; error 41 us; skew 500 ppm
Jun 19 02:34:55 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:55.879161 14134 init.cc:377] Logged in from keytab as oryx/127.7.194.193@KRBTEST.COM (short username oryx)
I20260619 02:34:55.880132 14134 webserver.cc:460] Webserver started at http://127.7.194.193:36371/ using document root <none> and password file <none>
I20260619 02:34:55.880688 14134 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:55.880751 14134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:55.880923 14134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:55.883164 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "baa5e2361000406e892e89dac775c4b7"
format_stamp: "Formatted at 2026-06-19 02:34:55 on dist-test-slave-9tkz"
server_key: "b08138c47cd86f855c065c9d9ea79ab9"
server_key_iv: "8afc63a399b042f659095d978f11904c"
server_key_version: "encryptionkey@0"
I20260619 02:34:55.883769 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "baa5e2361000406e892e89dac775c4b7"
format_stamp: "Formatted at 2026-06-19 02:34:55 on dist-test-slave-9tkz"
server_key: "b08138c47cd86f855c065c9d9ea79ab9"
server_key_iv: "8afc63a399b042f659095d978f11904c"
server_key_version: "encryptionkey@0"
I20260619 02:34:55.887739 14134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:34:55.890414 14149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:55.891928 14134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:55.892068 14134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "baa5e2361000406e892e89dac775c4b7"
format_stamp: "Formatted at 2026-06-19 02:34:55 on dist-test-slave-9tkz"
server_key: "b08138c47cd86f855c065c9d9ea79ab9"
server_key_iv: "8afc63a399b042f659095d978f11904c"
server_key_version: "encryptionkey@0"
I20260619 02:34:55.892194 14134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:55.912959 14134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:55.916324 14134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:55.916494 14134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:55.917171 14134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:55.918141 14134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:55.918226 14134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:55.918267 14134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:55.918283 14134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:55.927840 14134 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:42707
I20260619 02:34:55.927920 14262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:42707 every 8 connection(s)
I20260619 02:34:55.928862 14134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:55.930064 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 14134
I20260619 02:34:55.930197 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:34:55.930553 7947 external_mini_cluster.cc:1428] Setting key 9aab12ee56f245af762c76b7b48db093
Jun 19 02:34:55 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.7.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.194@KRBTEST.COM" created.
I20260619 02:34:55.945093 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:55.931115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:56607 (local address 127.7.194.254:45193)
0619 02:34:55.931261 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:55.931266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:55.932414 (+ 1148us) server_negotiation.cc:407] Connection header received
0619 02:34:55.934323 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:55.934327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:55.934393 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:55.934493 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:55.935878 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:55.936798 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:55.938159 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:55.938641 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:55.938781 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:55.941283 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:55.941290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:55.941296 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:55.941331 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:55.942908 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:55.943435 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:55.943437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:55.943439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:55.943514 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:55.943862 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:55.943864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:55.943866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:55.944093 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:55.944173 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:55.944779 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 02:34:55.944911 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:34:55.946054 14265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:55.931721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.193:56607)
0619 02:34:55.932250 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:55.932311 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:55.934064 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:55.934697 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:55.934705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:55.935075 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:55.935720 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:55.935736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:55.936941 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:55.936945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:55.938029 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:55.938039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:55.938797 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:55.938800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:55.938945 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:55.939596 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:55.939616 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:55.941097 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:55.943049 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:55.943058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:55.943078 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:55.943348 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:55.943593 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:55.943595 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:55.943596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:55.943753 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:55.944184 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:55.944189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:55.944476 (+ 287us) client_negotiation.cc:769] Sending connection context
0619 02:34:55.944737 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 02:34:55.945091 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260619 02:34:55.947574 14263 heartbeater.cc:347] Connected to a master server at 127.7.194.254:45193
I20260619 02:34:55.947835 14263 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:55.948352 14263 heartbeater.cc:511] Master 127.7.194.254:45193 requested a full tablet report, sending...
I20260619 02:34:55.950007 14072 ts_manager.cc:194] Registered new tserver with Master: baa5e2361000406e892e89dac775c4b7 (127.7.194.193:42707)
I20260619 02:34:55.951189 14072 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:56607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab.
Entry for principal oryx/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab.
I20260619 02:34:55.961153 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:45193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab
--principal=oryx/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:56.078311 14268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:56.078595 14268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:56.078639 14268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:56.082674 14268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:56.082736 14268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:56.082798 14268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:56.088251 14268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab
--principal=oryx/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:45193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:56.089318 14268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:56.090438 14268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:56.100344 14276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:56.100353 14274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:56.100535 14273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:56.100510 14268 server_base.cc:1034] running on GCE node
I20260619 02:34:56.101040 14268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:56.101727 14268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:56.102916 14268 hybrid_clock.cc:648] HybridClock initialized: now 1781836496102901 us; error 35 us; skew 500 ppm
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:56.105736 14268 init.cc:377] Logged in from keytab as oryx/127.7.194.194@KRBTEST.COM (short username oryx)
I20260619 02:34:56.106652 14268 webserver.cc:460] Webserver started at http://127.7.194.194:36081/ using document root <none> and password file <none>
I20260619 02:34:56.107302 14268 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:56.107393 14268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:56.107635 14268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:56.109823 14268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "b090dd45ea8e4f3294be929815ed3472"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "e69241e92db2d9a3bad642a19700b0e0"
server_key_iv: "c594fb35a76d00777b3f989927f80716"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.110419 14268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "b090dd45ea8e4f3294be929815ed3472"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "e69241e92db2d9a3bad642a19700b0e0"
server_key_iv: "c594fb35a76d00777b3f989927f80716"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.114396 14268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:56.117072 14283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.118530 14268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:56.118686 14268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "b090dd45ea8e4f3294be929815ed3472"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "e69241e92db2d9a3bad642a19700b0e0"
server_key_iv: "c594fb35a76d00777b3f989927f80716"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.118815 14268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:56.142086 14268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:56.145380 14268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:56.145660 14268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:56.146328 14268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:56.147202 14268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:56.147274 14268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.147347 14268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:56.147392 14268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.156505 14268 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:40645
I20260619 02:34:56.156549 14396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:40645 every 8 connection(s)
I20260619 02:34:56.157501 14268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:56.158248 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 14268
I20260619 02:34:56.158376 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:34:56.158737 7947 external_mini_cluster.cc:1428] Setting key ccb86bc30798f38990fc688bbd2a9aca
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.7.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.195@KRBTEST.COM" created.
I20260619 02:34:56.174314 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.159734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:57957 (local address 127.7.194.254:45193)
0619 02:34:56.159849 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.159854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.161259 (+ 1405us) server_negotiation.cc:407] Connection header received
0619 02:34:56.162980 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.162984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.163040 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.163126 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.164393 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.165273 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.166618 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.167055 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.167241 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.170167 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.170171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.170174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.170207 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.171670 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.172405 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.172408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.172410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.172493 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.172982 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.172988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.172993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.173251 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.173335 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.174057 (+ 722us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.174183 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:34:56.175414 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.160357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.194:57957)
0619 02:34:56.161075 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.161144 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.162741 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.163330 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.163338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.163726 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.164240 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.164254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.165436 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.165440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.166475 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.166485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.167284 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.167287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.167456 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.168205 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.168223 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.169958 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.171785 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.171791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.171806 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.172264 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.172627 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.172663 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.172668 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.172869 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.173363 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.173371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.173775 (+ 404us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.174001 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.174425 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":106,"threads_started":1}
I20260619 02:34:56.176676 14397 heartbeater.cc:347] Connected to a master server at 127.7.194.254:45193
I20260619 02:34:56.176942 14397 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:56.177470 14397 heartbeater.cc:511] Master 127.7.194.254:45193 requested a full tablet report, sending...
I20260619 02:34:56.178681 14072 ts_manager.cc:194] Registered new tserver with Master: b090dd45ea8e4f3294be929815ed3472 (127.7.194.194:40645)
I20260619 02:34:56.179284 14072 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:57957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab.
Entry for principal oryx/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab.
I20260619 02:34:56.189500 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:45193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab
--principal=oryx/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:56.303841 14402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:56.304148 14402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:56.304198 14402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:56.308298 14402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:56.308363 14402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:56.308423 14402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:56.313881 14402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab
--principal=oryx/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:45193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:56.315232 14402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:56.316430 14402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:56.326647 14408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:56.326592 14410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:56.326817 14407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:56.326740 14402 server_base.cc:1034] running on GCE node
I20260619 02:34:56.327306 14402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:56.327981 14402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:56.329162 14402 hybrid_clock.cc:648] HybridClock initialized: now 1781836496329126 us; error 49 us; skew 500 ppm
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:56.337415 14402 init.cc:377] Logged in from keytab as oryx/127.7.194.195@KRBTEST.COM (short username oryx)
I20260619 02:34:56.338515 14402 webserver.cc:460] Webserver started at http://127.7.194.195:41411/ using document root <none> and password file <none>
I20260619 02:34:56.339131 14402 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:56.339224 14402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:56.339465 14402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:56.341694 14402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "4ff80a9a53a84a5f9a8b60a334680d98"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "f05b3ab4750cb6cf4128af2a04d6e85b"
server_key_iv: "913c5bf3fafa909308eac02a35eb5913"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.342288 14402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "4ff80a9a53a84a5f9a8b60a334680d98"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "f05b3ab4750cb6cf4128af2a04d6e85b"
server_key_iv: "913c5bf3fafa909308eac02a35eb5913"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.346259 14402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:34:56.348762 14417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.350066 14402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:34:56.350205 14402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "4ff80a9a53a84a5f9a8b60a334680d98"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "f05b3ab4750cb6cf4128af2a04d6e85b"
server_key_iv: "913c5bf3fafa909308eac02a35eb5913"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.350335 14402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:56.375241 14402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:56.378798 14402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:56.379033 14402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:56.379714 14402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:56.380584 14402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:56.380656 14402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.380724 14402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:56.380770 14402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.390122 14402 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:40641
I20260619 02:34:56.390161 14530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:40641 every 8 connection(s)
I20260619 02:34:56.391193 14402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:56.396837 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 14402
I20260619 02:34:56.396991 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:34:56.397294 7947 external_mini_cluster.cc:1428] Setting key da71109e5f269ce56b0285002efcc271
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:34:56.405256 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.393192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:51859 (local address 127.7.194.254:45193)
0619 02:34:56.393333 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.393336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.394245 (+ 909us) server_negotiation.cc:407] Connection header received
0619 02:34:56.395582 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.395585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.395636 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.395714 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.397076 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.397780 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.398965 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.399207 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.399322 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.401902 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.401906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.401909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.401943 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.403191 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.403705 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.403708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.403712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.403799 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.404147 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.404152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.404153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.404352 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.404463 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.404962 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.405082 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:34:56.406176 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.393645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.195:51859)
0619 02:34:56.394086 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.394130 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.395371 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.395863 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.395870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.396333 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.396876 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.396891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.397910 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.397915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.398838 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.398847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.399359 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.399362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.399547 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.400193 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.400212 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.401743 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.403318 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.403324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.403340 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.403599 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.403899 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.403902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.403904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.404051 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.404459 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.404465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.404729 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.404924 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.405255 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260619 02:34:56.407298 14531 heartbeater.cc:347] Connected to a master server at 127.7.194.254:45193
I20260619 02:34:56.407521 14531 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:56.407982 14531 heartbeater.cc:511] Master 127.7.194.254:45193 requested a full tablet report, sending...
I20260619 02:34:56.408979 14072 ts_manager.cc:194] Registered new tserver with Master: 4ff80a9a53a84a5f9a8b60a334680d98 (127.7.194.195:40641)
I20260619 02:34:56.409747 14072 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:51859
I20260619 02:34:56.411196 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.194.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.253.keytab.
Entry for principal oryx/127.7.194.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.253.keytab.
I20260619 02:34:56.438689 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.253:35843
--webserver_interface=127.7.194.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.253.keytab
--principal=oryx/127.7.194.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:45193,127.7.194.253:35843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:34:56.554273 14536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:56.554641 14536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:56.554738 14536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:56.558941 14536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:56.559028 14536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:56.559063 14536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:56.559119 14536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:56.559157 14536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:56.564891 14536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:45193,127.7.194.253:35843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.253.keytab
--principal=oryx/127.7.194.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.253:35843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.194.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
W20260619 02:34:56.565183 14536 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.194.254:45193,127.7.194.253:35843'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 02:34:56.566231 14536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:56.567329 14536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:34:56.576411 14536 server_base.cc:1034] running on GCE node
W20260619 02:34:56.576366 14542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:56.576349 14544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:56.576344 14541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:56.577270 14536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:56.578493 14536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:56.579687 14536 hybrid_clock.cc:648] HybridClock initialized: now 1781836496579674 us; error 31 us; skew 500 ppm
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:56.583158 14536 init.cc:377] Logged in from keytab as oryx/127.7.194.253@KRBTEST.COM (short username oryx)
I20260619 02:34:56.584568 14536 webserver.cc:460] Webserver started at http://127.7.194.253:44947/ using document root <none> and password file <none>
I20260619 02:34:56.585394 14536 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:56.585484 14536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:56.585765 14536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:56.588510 14536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data/instance:
uuid: "905b3b91feb74d8ca2904dea1064bedd"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "2a2f196befcacd67007bd19c5bf64ed6"
server_key_iv: "5f3c9c1ac173e5725abbe06f0cbce3b3"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.589283 14536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal/instance:
uuid: "905b3b91feb74d8ca2904dea1064bedd"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "2a2f196befcacd67007bd19c5bf64ed6"
server_key_iv: "5f3c9c1ac173e5725abbe06f0cbce3b3"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.593779 14536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:34:56.596386 14551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.597822 14536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:56.597924 14536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal
uuid: "905b3b91feb74d8ca2904dea1064bedd"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "2a2f196befcacd67007bd19c5bf64ed6"
server_key_iv: "5f3c9c1ac173e5725abbe06f0cbce3b3"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.598018 14536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:56.615983 14536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:56.619314 14536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:56.619484 14536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:56.626854 14536 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.253:35843
I20260619 02:34:56.627144 14603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.253:35843 every 8 connection(s)
I20260619 02:34:56.627928 14536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
I20260619 02:34:56.630883 14604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:56.634827 14604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:34:56.635466 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 14536
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.253@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.194.253@KRBTEST.COM
I20260619 02:34:56.649685 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.637738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.7.194.254:45193)
0619 02:34:56.637859 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.637864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.638269 (+ 405us) server_negotiation.cc:407] Connection header received
0619 02:34:56.638705 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.638709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.638763 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.638874 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.640014 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.640537 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.641764 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.642112 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.642218 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.645250 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.645258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.645280 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.645311 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.647295 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.647845 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.647849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.647851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.647942 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.648353 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.648357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.648358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.648699 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.648802 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.649363 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.649503 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:34:56.650611 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.637668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:58458)
0619 02:34:56.638138 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.638186 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.638510 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.639051 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.639058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.639541 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.639880 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.639892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.640673 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.640678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.641642 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.641651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.642271 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.642275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.642475 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.643167 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.643184 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.645090 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.647410 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.647417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.647426 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.647743 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.648042 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.648048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.648050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.648232 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.648807 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.648812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.649103 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.649302 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.649684 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260619 02:34:56.650615 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.637017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.7.194.253:35843)
0619 02:34:56.637446 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.637463 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.637504 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:34:56.638419 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.638443 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.638704 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.638888 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.639817 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.641125 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.642195 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.642682 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.642795 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.645907 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.645914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.645929 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.645965 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.647859 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.648368 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.648374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.648376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.648447 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.648732 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.648738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.648743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.649093 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.649236 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.649521 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.649753 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":277,"spinlock_wait_cycles":49536,"thread_start_us":115,"threads_started":1}
I20260619 02:34:56.652719 14604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.253@KRBTEST.COM for oryx/127.7.194.253@KRBTEST.COM
I20260619 02:34:56.661348 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.653243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.0.0.1:34662)
0619 02:34:56.653371 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.653392 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.653456 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.653732 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.653734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.653908 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.653998 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.654003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.654745 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.654747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.655299 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.655307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.656177 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.656179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.656310 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.656805 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.656817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.658124 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.659970 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.659972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.659974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.660165 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.660417 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.660419 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.660421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.660491 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.660907 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.660909 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.660984 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.661067 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.661181 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":48}
I20260619 02:34:56.661358 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.653322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.7.194.253:35843)
0619 02:34:56.653429 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.653432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.653444 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:34:56.653513 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.653515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.653604 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.653693 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.654093 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.654588 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.655391 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.656083 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.656215 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.658264 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.658267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.658268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.658288 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.659872 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.660262 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.660270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.660271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.660341 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.660598 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.660601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.660603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.660803 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.660914 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.661124 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.661231 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":53}
I20260619 02:34:56.664327 14604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Bootstrap starting.
I20260619 02:34:56.666774 14604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:56.667461 14604 log.cc:824] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:56.669420 14604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: No bootstrap required, opened a new log
I20260619 02:34:56.672658 14604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } }
I20260619 02:34:56.672863 14604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:56.672902 14604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905b3b91feb74d8ca2904dea1064bedd, State: Initialized, Role: FOLLOWER
I20260619 02:34:56.673354 14604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } }
I20260619 02:34:56.673864 14610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } } }
I20260619 02:34:56.674008 14610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:34:56.674633 14604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:34:56.679502 14621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:34:56.679610 14621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:34:56.681331 14536 master_runner.cc:428] Detected that this master 905b3b91feb74d8ca2904dea1064bedd is joining an existing cluster
I20260619 02:34:56.681406 14536 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.194.253:35843
I20260619 02:34:56.692669 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.683468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:58470)
0619 02:34:56.683906 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.683920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.684058 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.684360 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.684363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.684611 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.684784 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.684792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.685693 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.685699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.686554 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.686563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.687146 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.687149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.687228 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.687764 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.687775 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.688206 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.690236 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.690503 (+ 267us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.690504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.690807 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.691174 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.691178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.691180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.691270 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.692089 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.692092 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.692196 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.692295 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.692431 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":23,"thread_start_us":111,"threads_started":1}
I20260619 02:34:56.692714 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.683597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.7.194.254:45193)
0619 02:34:56.683735 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.683739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.683987 (+ 248us) server_negotiation.cc:407] Connection header received
0619 02:34:56.684176 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.684180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.684240 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.684331 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.684909 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.685530 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.686679 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.687020 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.687135 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.688325 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.688332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.688335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.688364 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.690119 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.690924 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.690928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.690930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.691024 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.691375 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.691378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.691380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.691602 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.691676 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.692355 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.692494 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:34:56.693537 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.683960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.7.194.253:35843)
0619 02:34:56.684048 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.684052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.684406 (+ 354us) server_negotiation.cc:407] Connection header received
0619 02:34:56.684545 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.684549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.684595 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.684677 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.685177 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.685959 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.686933 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.687232 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.687332 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.689725 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.689729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.689731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.689757 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.691367 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.692063 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.692066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.692067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.692143 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.692463 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.692466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.692469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.692677 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.692982 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.693265 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.693413 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":27}
I20260619 02:34:56.693572 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.683914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.0.0.1:34676)
0619 02:34:56.684338 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.684352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.684455 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.684691 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.684693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.684939 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.685048 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.685054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.686082 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.686085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.686832 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.686838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.687332 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.687335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.687809 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.688666 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.688680 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.689538 (+ 858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.691721 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.691724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.691726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.691958 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.692273 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.692278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.692280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.692366 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.693040 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.693041 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.693117 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.693223 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.693360 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":22,"thread_start_us":189,"threads_started":1}
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.254@KRBTEST.COM for oryx/127.7.194.253@KRBTEST.COM
I20260619 02:34:56.708663 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.697866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.0.0.1:34686)
0619 02:34:56.698289 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.698351 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.698488 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.698809 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.698816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.699296 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.699429 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.699436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.700203 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.700207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.701268 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.701279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.701840 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.701844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.701956 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.702481 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.702495 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.704452 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.706593 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.706603 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.706610 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.706926 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.707254 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.707258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.707260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.707355 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.707914 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.707922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.708071 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.708235 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.708442 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":118,"threads_started":1}
I20260619 02:34:56.709028 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.697891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.7.194.253:35843)
0619 02:34:56.698027 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.698031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.698436 (+ 405us) server_negotiation.cc:407] Connection header received
0619 02:34:56.698620 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.698622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.698665 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.698760 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.699569 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.700091 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.701408 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.701732 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.701814 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.704633 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.704636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.704638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.704659 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.706468 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.707042 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.707048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.707051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.707149 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.707521 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.707527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.707532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.707797 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.707902 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.708313 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.708450 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:34:56.710516 14072 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.194.253:35843: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd"
member_type: NON_VOTER
last_known_addr {
host: "127.7.194.253"
port: 35843
}
attrs {
promote: true
}
}
dest_uuid: "71ef3db455d5435da039949bfaa1ac86"
cas_config_opid_index: -1
I20260619 02:34:56.711237 14072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.7.194.252@KRBTEST.COM; defaulting to no policy
I20260619 02:34:56.713294 14633 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 905b3b91feb74d8ca2904dea1064bedd (127.7.194.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } }
I20260619 02:34:56.714186 14578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:56.715210 14633 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.194.253:35843
I20260619 02:34:56.715651 14536 master.cc:537] Master@127.7.194.253:35843 shutting down...
I20260619 02:34:56.715708 14578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Refusing update from remote peer 71ef3db455d5435da039949bfaa1ac86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 02:34:56.715729 14632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } }
I20260619 02:34:56.716537 14632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:56.716419 14633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.252@KRBTEST.COM" created.
W20260619 02:34:56.720399 14058 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 02:34:56.721215 14536 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 02:34:56.721311 14058 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 -> Peer 905b3b91feb74d8ca2904dea1064bedd (127.7.194.253:35843): Couldn't send request to peer 905b3b91feb74d8ca2904dea1064bedd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:34:56.721506 14536 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:56.721698 14536 tablet_replica.cc:330] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.252.keytab.
Entry for principal oryx/127.7.194.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.252.keytab.
I20260619 02:34:56.736474 14536 master.cc:552] Master@127.7.194.253:35843 shutdown complete.
I20260619 02:34:56.736470 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.252:37941
--webserver_interface=127.7.194.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.252.keytab
--principal=oryx/127.7.194.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:45193,127.7.194.253:35843,127.7.194.252:37941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:34:56.736538 14536 master_runner.cc:305] Clearing existing system tablet
I20260619 02:34:56.736853 14536 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:56.736907 14536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:56.738339 14536 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:34:56.739282 14639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.739552 14536 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:56.739676 14536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:56.739709 14536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal
uuid: "905b3b91feb74d8ca2904dea1064bedd"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "2a2f196befcacd67007bd19c5bf64ed6"
server_key_iv: "5f3c9c1ac173e5725abbe06f0cbce3b3"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.741978 14536 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:56.744421 14536 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:34:56.744510 14536 log.cc:1196] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 02:34:56.744769 14536 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Deleting consensus metadata
I20260619 02:34:56.744935 14536 master_runner.cc:315] Copying system tablet from 127.7.194.254:45193
I20260619 02:34:56.746188 14536 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: tablet copy: Beginning tablet copy session from remote peer at address 127.7.194.254:45193
I20260619 02:34:56.755681 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.747123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.7.194.254:45193)
0619 02:34:56.747237 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.747242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.747259 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:34:56.747314 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.747317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.747365 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.747456 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.748012 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.748834 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.750086 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.750457 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.750562 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.751892 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.751896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.751900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.751928 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.753705 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.754289 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.754293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.754295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.754384 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.754670 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.754674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.754676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.754898 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.755020 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.755373 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.755501 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:34:56.756245 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.746741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:58482)
0619 02:34:56.747066 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.747087 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.747225 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.747500 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.747505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.747735 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.747875 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.747882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.748951 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.748954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.749943 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.749953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.750574 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.750577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.750695 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.751299 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.751314 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.751741 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.753821 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.753827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.753831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.754137 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.754478 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.754481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.754483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.754578 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.755076 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.755079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.755182 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.755894 (+ 712us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.756043 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260619 02:34:56.756810 14092 tablet_copy_service.cc:140] P 71ef3db455d5435da039949bfaa1ac86: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 905b3b91feb74d8ca2904dea1064bedd ({username='oryx', principal='oryx/127.7.194.253@KRBTEST.COM'} at 127.0.0.1:58482)
I20260619 02:34:56.757004 14092 tablet_copy_service.cc:161] P 71ef3db455d5435da039949bfaa1ac86: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 905b3b91feb74d8ca2904dea1064bedd at {username='oryx', principal='oryx/127.7.194.253@KRBTEST.COM'} at 127.0.0.1:58482: session id = 905b3b91feb74d8ca2904dea1064bedd-00000000000000000000000000000000
I20260619 02:34:56.758970 14092 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:34:56.760964 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:34:56.763410 14536 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: tablet copy: Starting download of 0 data blocks...
I20260619 02:34:56.763643 14536 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: tablet copy: Starting download of 1 WAL segments...
I20260619 02:34:56.765502 14536 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:34:56.767336 14092 tablet_copy_service.cc:342] P 71ef3db455d5435da039949bfaa1ac86: Request end of tablet copy session 905b3b91feb74d8ca2904dea1064bedd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.194.253@KRBTEST.COM'} at 127.0.0.1:58482
I20260619 02:34:56.767454 14092 tablet_copy_service.cc:433] P 71ef3db455d5435da039949bfaa1ac86: ending tablet copy session 905b3b91feb74d8ca2904dea1064bedd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 905b3b91feb74d8ca2904dea1064bedd
I20260619 02:34:56.775059 14536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:56.776870 14646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:56.777040 14647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:56.777034 14649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:56.777648 14536 server_base.cc:1034] running on GCE node
I20260619 02:34:56.777844 14536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:56.778198 14536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:56.779403 14536 hybrid_clock.cc:648] HybridClock initialized: now 1781836496779386 us; error 34 us; skew 500 ppm
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:56.781173 14536 init.cc:377] Logged in from keytab as oryx/127.7.194.253@KRBTEST.COM (short username oryx)
I20260619 02:34:56.781740 14536 webserver.cc:460] Webserver started at http://127.7.194.253:39277/ using document root <none> and password file <none>
I20260619 02:34:56.781975 14536 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:56.782076 14536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:56.783541 14536 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:56.784593 14656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.784861 14536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:56.784974 14536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal
uuid: "905b3b91feb74d8ca2904dea1064bedd"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "2a2f196befcacd67007bd19c5bf64ed6"
server_key_iv: "5f3c9c1ac173e5725abbe06f0cbce3b3"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.785127 14536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:56.807605 14536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:56.810058 14536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:56.814842 14536 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.253:35843
I20260619 02:34:56.814884 14708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.253:35843 every 8 connection(s)
I20260619 02:34:56.815629 14536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
I20260619 02:34:56.817673 14709 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:34:56.818773 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Bootstrap starting.
I20260619 02:34:56.829895 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:34:56.830266 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Bootstrap complete.
I20260619 02:34:56.830874 14709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } }
I20260619 02:34:56.831008 14709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 905b3b91feb74d8ca2904dea1064bedd, State: Initialized, Role: LEARNER
I20260619 02:34:56.831115 14709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } }
I20260619 02:34:56.831555 14709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:56.831480 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } }
I20260619 02:34:56.831735 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: This master's current role is: LEARNER
I20260619 02:34:56.834403 14723 catalog_manager.cc:1259] Loaded cluster ID: e6c6f9ee024b4211a751114d0a536d6e
I20260619 02:34:56.834506 14723 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: loading cluster ID for follower catalog manager: success
I20260619 02:34:56.836637 14723 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: acquiring CA information for follower catalog manager: success
I20260619 02:34:56.838187 14723 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:34:56.879081 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:56.879388 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:56.879474 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:56.883785 14637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:34:56.883852 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:56.883873 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:56.883891 14637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:34:56.883906 14637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:34:56.889501 14637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:45193,127.7.194.253:35843,127.7.194.252:37941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.252.keytab
--principal=oryx/127.7.194.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.252:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.194.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:56.890836 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:56.891978 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:56.901520 14728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:56.901701 14637 server_base.cc:1034] running on GCE node
W20260619 02:34:56.901631 14729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:56.901520 14731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:56.902426 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:56.903673 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:56.904835 14637 hybrid_clock.cc:648] HybridClock initialized: now 1781836496904820 us; error 32 us; skew 500 ppm
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:56.907763 14637 init.cc:377] Logged in from keytab as oryx/127.7.194.252@KRBTEST.COM (short username oryx)
I20260619 02:34:56.908710 14637 webserver.cc:460] Webserver started at http://127.7.194.252:42121/ using document root <none> and password file <none>
I20260619 02:34:56.909273 14637 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:56.909334 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:56.909605 14637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:34:56.911759 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data/instance:
uuid: "91e26bb975234ecaadc52946641ecd68"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "a896ad484acfd48eff49f07553b6c7f8"
server_key_iv: "36d51c96de4509c2e24f493f67747fc0"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.912341 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal/instance:
uuid: "91e26bb975234ecaadc52946641ecd68"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "a896ad484acfd48eff49f07553b6c7f8"
server_key_iv: "36d51c96de4509c2e24f493f67747fc0"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.916397 14637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 02:34:56.919028 14738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:56.920461 14637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:34:56.920617 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal
uuid: "91e26bb975234ecaadc52946641ecd68"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "a896ad484acfd48eff49f07553b6c7f8"
server_key_iv: "36d51c96de4509c2e24f493f67747fc0"
server_key_version: "encryptionkey@0"
I20260619 02:34:56.920739 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:56.941195 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:56.944528 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:56.944763 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:56.952625 14637 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.252:37941
I20260619 02:34:56.953016 14790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.252:37941 every 8 connection(s)
I20260619 02:34:56.953743 14263 heartbeater.cc:503] Master 127.7.194.254:45193 was elected leader, sending a full tablet report...
I20260619 02:34:56.953922 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
I20260619 02:34:56.954329 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 14637
I20260619 02:34:56.957481 14791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:56.962566 14791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.194.252@KRBTEST.COM
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.252@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:34:56.971024 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.955980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.7.194.252:37941)
0619 02:34:56.956388 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.956407 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.956451 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:34:56.957328 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.957353 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.957864 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.958251 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.958981 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.960607 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.961706 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.962400 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.962570 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.965328 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.965334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.965350 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.965381 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.967417 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.967885 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.967889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.967894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.967999 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.968288 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.968291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.968293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.968847 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.969005 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.969709 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.970027 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260619 02:34:56.974507 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.964621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.7.194.254:45193)
0619 02:34:56.964776 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.964780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.965144 (+ 364us) server_negotiation.cc:407] Connection header received
0619 02:34:56.965328 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.965332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.965386 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.965482 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.966179 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.967006 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.968157 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.968455 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.968566 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.971038 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.971042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.971044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.971119 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.972747 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.973174 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.973178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.973180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.973273 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.973599 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.973602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.973604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.973898 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.973982 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.974297 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.974394 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:34:56.974527 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.964489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:58488)
0619 02:34:56.964999 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.965056 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.965188 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.965514 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.965518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.965934 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.966048 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.966054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.967150 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.967153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.968017 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.968026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.968645 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.968649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.968727 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.969249 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.969263 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.970895 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.972847 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.972854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.972858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.973065 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.973362 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.973364 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.973365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.973486 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.973999 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.974004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.974117 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.974226 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.974368 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":185,"threads_started":1}
I20260619 02:34:56.975564 14791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.252@KRBTEST.COM for oryx/127.7.194.253@KRBTEST.COM
I20260619 02:34:56.985095 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.976053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.0.0.1:34700)
0619 02:34:56.976414 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.976426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.976486 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.976721 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.976723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.976858 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.976949 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.976953 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.978005 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.978009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.979044 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.979056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.979543 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.979547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.979695 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.980200 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.980222 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.981701 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.983574 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.983577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.983580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.983785 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.984087 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.984091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.984093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.984181 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.984559 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.984562 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.984670 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.984775 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.984917 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299}
I20260619 02:34:56.985122 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.976200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.7.194.253:35843)
0619 02:34:56.976451 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.976455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.976467 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:34:56.976580 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.976583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.976627 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.976695 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.977041 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.977893 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.979176 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.979424 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.979503 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.981846 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.981850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.981852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.981896 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.983454 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.983883 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.983888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.983890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.983994 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.984269 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.984274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.984276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.984467 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.984570 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.984837 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.984958 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20260619 02:34:56.986303 14791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:34:56 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.252@KRBTEST.COM for oryx/127.7.194.252@KRBTEST.COM
I20260619 02:34:56.994303 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.986824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:37941 (local address 127.0.0.1:36516)
0619 02:34:56.986944 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:56.986960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:56.987038 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:56.987230 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:56.987232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:56.987358 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:56.987429 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.987433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.988478 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.988481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.989037 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:56.989045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.989450 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.989453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:56.989511 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.989968 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:56.989979 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:56.991148 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:56.992905 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.992908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.992909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.993093 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.993320 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:56.993322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:56.993323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:56.993393 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:56.993889 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:56.993892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:56.993974 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 02:34:56.994070 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:34:56.994177 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:34:56.994387 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:56.986894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.7.194.252:37941)
0619 02:34:56.987004 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:56.987007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:56.987019 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:34:56.987107 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:56.987109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:56.987142 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:56.987204 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:56.987516 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.988018 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.989143 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:56.989359 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:56.989437 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:56.991316 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:56.991322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:56.991327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:56.991357 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:56.992816 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.993174 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.993177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.993178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.993244 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:56.993495 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:56.993498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:56.993500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:56.993758 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:56.993865 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:56.994149 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 02:34:56.994247 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:34:56.997463 14791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: Bootstrap starting.
I20260619 02:34:56.999925 14791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:57.001137 14791 log.cc:824] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:57.003317 14791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: No bootstrap required, opened a new log
I20260619 02:34:57.006418 14791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } }
I20260619 02:34:57.006649 14791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:57.006736 14791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e26bb975234ecaadc52946641ecd68, State: Initialized, Role: FOLLOWER
I20260619 02:34:57.007222 14791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } }
I20260619 02:34:57.007740 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } } }
I20260619 02:34:57.007866 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:34:57.008483 14791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:34:57.013952 14808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:34:57.014040 14808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:34:57.017382 14637 master_runner.cc:428] Detected that this master 91e26bb975234ecaadc52946641ecd68 is joining an existing cluster
I20260619 02:34:57.017436 14637 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.194.252:37941
I20260619 02:34:57.021322 14797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:57.021483 14797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } }
I20260619 02:34:57.022531 14797 leader_election.cc:288] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ef3db455d5435da039949bfaa1ac86 (127.7.194.254:45193), 905b3b91feb74d8ca2904dea1064bedd (127.7.194.253:35843)
I20260619 02:34:57.023527 14082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91e26bb975234ecaadc52946641ecd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ef3db455d5435da039949bfaa1ac86" is_pre_election: true
I20260619 02:34:57.023710 14683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91e26bb975234ecaadc52946641ecd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905b3b91feb74d8ca2904dea1064bedd" is_pre_election: true
I20260619 02:34:57.023890 14683 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 LEARNER]: Handling vote request from an unknown peer 91e26bb975234ecaadc52946641ecd68
I20260619 02:34:57.024040 14683 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 91e26bb975234ecaadc52946641ecd68 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260619 02:34:57.025192 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 14134
I20260619 02:34:57.027261 14741 leader_election.cc:302] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91e26bb975234ecaadc52946641ecd68; no voters: 71ef3db455d5435da039949bfaa1ac86, 905b3b91feb74d8ca2904dea1064bedd
I20260619 02:34:57.027859 14797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:57.029489 14797 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 02:34:57.033026 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:42707
--local_ip_for_outbound_sockets=127.7.194.193
--tserver_master_addrs=127.7.194.254:45193
--webserver_port=36371
--webserver_interface=127.7.194.193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab
--principal=oryx/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.194.254:45193,127.7.194.253:35843
--tserver_master_addrs=127.7.194.254:45193,127.7.194.253:35843,127.7.194.252:37941 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:34:57.047596 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.020302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.7.194.252:37941)
0619 02:34:57.020414 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.020418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.020939 (+ 521us) server_negotiation.cc:407] Connection header received
0619 02:34:57.021069 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.021072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.021116 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.021165 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.021739 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.022436 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.023445 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.023725 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.023827 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.025578 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.025582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.025584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.025603 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.027320 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.027760 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.027764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.027765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.027832 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.028053 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.028056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.028057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.045956 (+ 17899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.046121 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.047247 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.047383 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:34:57.048178 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.020273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:37941 (local address 127.0.0.1:36518)
0619 02:34:57.020874 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.020889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.020986 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.021307 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.021310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.021478 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.021632 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.021640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.022556 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.022559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.023323 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.023332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.023948 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.023951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.024037 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.024933 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.024947 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.025423 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.027412 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.027415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.027417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.027663 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.027914 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.027916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.027918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.027981 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.046170 (+ 18189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.046176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.046316 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.047828 (+ 1512us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.047986 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":93,"threads_started":1}
I20260619 02:34:57.051050 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.019891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.0.0.1:34704)
0619 02:34:57.020555 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.020575 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.020698 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.021027 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.021031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.021608 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.022018 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.022026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.024377 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.024380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.025956 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.025964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.029036 (+ 3072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.029039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.029123 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.046258 (+ 17135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.046275 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.046814 (+ 539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.049398 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.049401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.049403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.049683 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.049984 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.049987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.049989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.050078 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.050548 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.050551 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.050639 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.050722 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.050857 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":88,"threads_started":1}
I20260619 02:34:57.051822 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.020013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.7.194.253:35843)
0619 02:34:57.020148 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.020154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.020653 (+ 499us) server_negotiation.cc:407] Connection header received
0619 02:34:57.020776 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.020778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.020833 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.020910 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.022216 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.023031 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.028476 (+ 5445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.028832 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.028928 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.047011 (+ 18083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.047016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.047019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.047060 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.049262 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.049799 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.049803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.049805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.049888 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.050173 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.050177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.050179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.050415 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.050504 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.051533 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.051670 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:34:57.052659 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.019587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.7.194.254:45193)
0619 02:34:57.019733 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.019738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.020416 (+ 678us) server_negotiation.cc:407] Connection header received
0619 02:34:57.020599 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.020602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.020657 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.020713 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.026254 (+ 5541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.027084 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.028280 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.028606 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.028696 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.047458 (+ 18762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.047463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.047466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.047498 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.050925 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.051414 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.051418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.051420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.051505 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.051781 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.051784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.051786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.051986 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.052064 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.052384 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.052503 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:34:57.053171 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.019495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:58500)
0619 02:34:57.020230 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.020245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.020488 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.020866 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.020868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.021307 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.026125 (+ 4818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.026133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.027237 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.027240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.028154 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.028160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.028812 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.028815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.028928 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.046600 (+ 17672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.046614 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.047073 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.051067 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.051070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.051072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.051299 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.051611 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.051614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.051616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.051689 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.052132 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.052134 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.052219 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.052852 (+ 633us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.052990 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":106,"threads_started":1}
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.254@KRBTEST.COM for oryx/127.7.194.252@KRBTEST.COM
I20260619 02:34:57.067351 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.057131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:37941 (local address 127.0.0.1:36534)
0619 02:34:57.057297 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.057310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.057382 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.057664 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.057669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.057859 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.057965 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.057974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.058935 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.058938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.059871 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.059882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.060408 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.060413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.060518 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.062093 (+ 1575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.062108 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.063681 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.065712 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.065716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.065718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.065973 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.066278 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.066281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.066284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.066366 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.066793 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.066796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.066893 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.067018 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.067177 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 02:34:57.067823 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.057210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.7.194.252:37941)
0619 02:34:57.057329 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.057335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.057349 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:57.057487 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.057489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.057565 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.057644 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.058088 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.058820 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.059981 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.060293 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.060396 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.063860 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.063864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.063867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.063893 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.065587 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.066096 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.066099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.066101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.066182 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.066459 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.066464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.066467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.066667 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.066751 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.067573 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.067687 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:34:57.068723 14072 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.194.252:37941: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "91e26bb975234ecaadc52946641ecd68"
member_type: NON_VOTER
last_known_addr {
host: "127.7.194.252"
port: 37941
}
attrs {
promote: true
}
}
dest_uuid: "71ef3db455d5435da039949bfaa1ac86"
cas_config_opid_index: 5
I20260619 02:34:57.069329 14072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } }
I20260619 02:34:57.070559 14632 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 91e26bb975234ecaadc52946641ecd68 (127.7.194.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.070977 14765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 FOLLOWER]: Refusing update from remote peer 71ef3db455d5435da039949bfaa1ac86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:34:57.071786 14633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:57.073663 14632 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.194.252:37941
I20260619 02:34:57.073900 14632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.074055 14632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:57.074687 14637 master.cc:537] Master@127.7.194.252:37941 shutting down...
I20260619 02:34:57.076922 14765 raft_consensus.cc:858] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } }
W20260619 02:34:57.077028 14765 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } }
I20260619 02:34:57.079985 14765 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 905b3b91feb74d8ca2904dea1064bedd (127.7.194.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } }
I20260619 02:34:57.082897 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ef3db455d5435da039949bfaa1ac86. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } }
I20260619 02:34:57.083029 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 02:34:57.083129 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } }
I20260619 02:34:57.083213 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 02:34:57.083294 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } }
I20260619 02:34:57.083366 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 02:34:57.084159 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.071589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.0.0.1:34718)
0619 02:34:57.074693 (+ 3104us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.074721 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.074809 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.075822 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.075825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.075846 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:57.075942 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.075948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.077213 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.077218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.078985 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.078996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.083689 (+ 4693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.083692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.083793 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.083844 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.083917 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.084006 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2999}
I20260619 02:34:57.084903 14633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.085039 14633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:57.085884 14637 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 02:34:57.086158 14637 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 02:34:57.086280 14637 tablet_replica.cc:330] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: stopping tablet replica
W20260619 02:34:57.086266 14058 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 -> Peer 91e26bb975234ecaadc52946641ecd68 (127.7.194.252:37941): Couldn't send request to peer 91e26bb975234ecaadc52946641ecd68. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:34:57.086382 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.071695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.7.194.253:35843)
0619 02:34:57.071825 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.071829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.075048 (+ 3219us) server_negotiation.cc:407] Connection header received
0619 02:34:57.075125 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.075128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.075190 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.075261 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:57.076088 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.077011 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.082652 (+ 5641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.083371 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.083483 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.083625 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.085434 (+ 1809us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.085531 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:34:57.086814 14683 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 LEARNER]: Refusing update from remote peer 71ef3db455d5435da039949bfaa1ac86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:34:57.087280 14632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 02:34:57.089089 14683 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 91e26bb975234ecaadc52946641ecd68 (127.7.194.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.091451 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ef3db455d5435da039949bfaa1ac86. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.091579 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: This master's current role is: LEARNER
I20260619 02:34:57.091665 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: NON_VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: true } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.091753 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: This master's current role is: LEARNER
I20260619 02:34:57.092329 14632 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: attempting to promote NON_VOTER 905b3b91feb74d8ca2904dea1064bedd to VOTER
I20260619 02:34:57.092746 14632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } }
I20260619 02:34:57.094583 14683 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 LEARNER]: Refusing update from remote peer 71ef3db455d5435da039949bfaa1ac86: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 02:34:57.095293 14633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 02:34:57.096968 14058 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 -> Peer 91e26bb975234ecaadc52946641ecd68 (127.7.194.252:37941): Couldn't send request to peer 91e26bb975234ecaadc52946641ecd68. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:34:57.098084 14633 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 905b3b91feb74d8ca2904dea1064bedd (127.7.194.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.099527 14633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.099619 14633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:57.099740 14633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.099803 14633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:57.100803 14683 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 905b3b91feb74d8ca2904dea1064bedd (127.7.194.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.103163 14823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.103273 14823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:34:57.114840 14637 master.cc:552] Master@127.7.194.252:37941 shutdown complete.
I20260619 02:34:57.114909 14637 master_runner.cc:305] Clearing existing system tablet
I20260619 02:34:57.115172 14637 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:57.115231 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:57.116602 14637 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:34:57.117641 14829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.117915 14637 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:57.118232 14637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:57.118335 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal
uuid: "91e26bb975234ecaadc52946641ecd68"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "a896ad484acfd48eff49f07553b6c7f8"
server_key_iv: "36d51c96de4509c2e24f493f67747fc0"
server_key_version: "encryptionkey@0"
I20260619 02:34:57.120154 14637 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:57.122632 14637 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:34:57.122720 14637 log.cc:1196] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 02:34:57.123037 14637 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: Deleting consensus metadata
I20260619 02:34:57.123206 14637 master_runner.cc:315] Copying system tablet from 127.7.194.254:45193
I20260619 02:34:57.124338 14637 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: tablet copy: Beginning tablet copy session from remote peer at address 127.7.194.254:45193
I20260619 02:34:57.134040 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.125059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.7.194.254:45193)
0619 02:34:57.125215 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.125237 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.125423 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:34:57.125555 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.125559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.125615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.125720 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.126248 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.127051 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.128216 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.128527 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.128607 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.130176 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.130180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.130189 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.130217 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.131982 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.132579 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.132582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.132584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.132679 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.132980 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.132983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.132985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.133217 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.133291 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.133706 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.133845 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:34:57.134869 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.124935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:58504)
0619 02:34:57.125315 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.125330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.125452 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.125756 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.125759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.125978 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.126113 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.126121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.127184 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.127188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.128074 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.128085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.128773 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.128776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.128866 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.129486 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.129500 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.129966 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.132130 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.132134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.132136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.132451 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.132791 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.132795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.132798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.132883 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.133362 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.133366 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.133471 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.134549 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.134707 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":102,"threads_started":1}
I20260619 02:34:57.135746 14092 tablet_copy_service.cc:140] P 71ef3db455d5435da039949bfaa1ac86: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 91e26bb975234ecaadc52946641ecd68 ({username='oryx', principal='oryx/127.7.194.252@KRBTEST.COM'} at 127.0.0.1:58504)
I20260619 02:34:57.135843 14092 tablet_copy_service.cc:161] P 71ef3db455d5435da039949bfaa1ac86: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 91e26bb975234ecaadc52946641ecd68 at {username='oryx', principal='oryx/127.7.194.252@KRBTEST.COM'} at 127.0.0.1:58504: session id = 91e26bb975234ecaadc52946641ecd68-00000000000000000000000000000000
I20260619 02:34:57.136953 14092 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:34:57.138538 14637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:34:57.141134 14637 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: tablet copy: Starting download of 0 data blocks...
I20260619 02:34:57.141343 14637 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: tablet copy: Starting download of 1 WAL segments...
I20260619 02:34:57.143047 14637 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:34:57.145231 14092 tablet_copy_service.cc:342] P 71ef3db455d5435da039949bfaa1ac86: Request end of tablet copy session 91e26bb975234ecaadc52946641ecd68-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.194.252@KRBTEST.COM'} at 127.0.0.1:58504
I20260619 02:34:57.145344 14092 tablet_copy_service.cc:433] P 71ef3db455d5435da039949bfaa1ac86: ending tablet copy session 91e26bb975234ecaadc52946641ecd68-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 91e26bb975234ecaadc52946641ecd68
I20260619 02:34:57.153952 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:57.155917 14836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:57.155936 14839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:57.157716 14837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:57.157730 14637 server_base.cc:1034] running on GCE node
I20260619 02:34:57.157989 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:57.158309 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:57.159456 14637 hybrid_clock.cc:648] HybridClock initialized: now 1781836497159442 us; error 32 us; skew 500 ppm
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:57.161204 14637 init.cc:377] Logged in from keytab as oryx/127.7.194.252@KRBTEST.COM (short username oryx)
I20260619 02:34:57.161712 14637 webserver.cc:460] Webserver started at http://127.7.194.252:37843/ using document root <none> and password file <none>
I20260619 02:34:57.161914 14637 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:57.161966 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:57.163522 14637 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:57.164618 14846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.164956 14637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:57.165048 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal
uuid: "91e26bb975234ecaadc52946641ecd68"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "a896ad484acfd48eff49f07553b6c7f8"
server_key_iv: "36d51c96de4509c2e24f493f67747fc0"
server_key_version: "encryptionkey@0"
I20260619 02:34:57.165136 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:57.172642 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:57.174870 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:57.179636 14637 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.252:37941
I20260619 02:34:57.180243 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
I20260619 02:34:57.180702 14898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.252:37941 every 8 connection(s)
I20260619 02:34:57.181524 14397 heartbeater.cc:503] Master 127.7.194.254:45193 was elected leader, sending a full tablet report...
I20260619 02:34:57.182299 14899 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:34:57.183497 14899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: Bootstrap starting.
W20260619 02:34:57.190718 14817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:57.191114 14817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:57.191192 14817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:34:57.192898 14899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:34:57.193230 14899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: Bootstrap complete.
I20260619 02:34:57.193856 14899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } }
I20260619 02:34:57.193994 14899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91e26bb975234ecaadc52946641ecd68, State: Initialized, Role: LEARNER
I20260619 02:34:57.194103 14899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } }
I20260619 02:34:57.194473 14905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.194572 14899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:34:57.194697 14905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: This master's current role is: LEARNER
W20260619 02:34:57.198438 14817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:57.198537 14817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:57.198657 14817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:34:57.198669 14916 catalog_manager.cc:1259] Loaded cluster ID: e6c6f9ee024b4211a751114d0a536d6e
I20260619 02:34:57.198715 14916 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: loading cluster ID for follower catalog manager: success
I20260619 02:34:57.200846 14916 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: acquiring CA information for follower catalog manager: success
I20260619 02:34:57.202260 14916 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:34:57.204567 14817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab
--principal=oryx/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:42707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=36371
--tserver_master_addrs=127.7.194.254:45193,127.7.194.253:35843,127.7.194.252:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:57.205766 14817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:57.206844 14817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:57.217110 14918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:57.217128 14919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:57.217293 14817 server_base.cc:1034] running on GCE node
W20260619 02:34:57.217569 14921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:57.218035 14817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:57.218745 14817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:57.219919 14817 hybrid_clock.cc:648] HybridClock initialized: now 1781836497219896 us; error 39 us; skew 500 ppm
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:57.222815 14817 init.cc:377] Logged in from keytab as oryx/127.7.194.193@KRBTEST.COM (short username oryx)
I20260619 02:34:57.223770 14817 webserver.cc:460] Webserver started at http://127.7.194.193:36371/ using document root <none> and password file <none>
I20260619 02:34:57.224340 14817 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:57.224436 14817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:57.228379 14817 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260619 02:34:57.230547 14928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.232029 14817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:57.232234 14817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "baa5e2361000406e892e89dac775c4b7"
format_stamp: "Formatted at 2026-06-19 02:34:55 on dist-test-slave-9tkz"
server_key: "b08138c47cd86f855c065c9d9ea79ab9"
server_key_iv: "8afc63a399b042f659095d978f11904c"
server_key_version: "encryptionkey@0"
I20260619 02:34:57.232875 14817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:57.245955 14817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:57.248978 14817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:57.249193 14817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:57.249886 14817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:57.250798 14817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:57.250872 14817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.250939 14817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:57.250991 14817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.260972 14817 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:42707
I20260619 02:34:57.260993 15041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:42707 every 8 connection(s)
I20260619 02:34:57.262065 14817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:34:57.271092 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 14817
I20260619 02:34:57.271337 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 14268
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.252@KRBTEST.COM
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.253@KRBTEST.COM
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:34:57.280203 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.266447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:36383 (local address 127.7.194.253:35843)
0619 02:34:57.266579 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.266583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.266595 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:34:57.266635 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.266637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.266682 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.266760 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.268200 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.269001 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.270233 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.270591 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.270665 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.275315 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.275320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.275323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.275354 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.277275 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.278051 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.278054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.278056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.278181 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.278658 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.278662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.278664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.278895 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.278963 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.279886 (+ 923us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.280030 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:34:57.280258 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.263897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:34451 (local address 127.7.194.254:45193)
0619 02:34:57.264045 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.264049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.265345 (+ 1296us) server_negotiation.cc:407] Connection header received
0619 02:34:57.266595 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.266598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.266670 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.266735 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.268195 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.268727 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.269977 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.270209 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.270336 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.275703 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.275707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.275709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.275768 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.277946 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.278658 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.278662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.278664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.279025 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.279358 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.279361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.279363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.279566 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.279652 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.279995 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.280109 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:34:57.280277 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.265919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:60913 (local address 127.7.194.252:37941)
0619 02:34:57.266201 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.266215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.266231 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:34:57.266579 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.266583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.266640 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.266705 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.268198 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.269234 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.270382 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.270722 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.270798 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.275361 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.275367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.275373 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.275408 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.278008 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.278765 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.278771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.278775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.278866 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.279207 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.279213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.279218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.279468 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.279538 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.279928 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.280062 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260619 02:34:57.280328 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:40645
--local_ip_for_outbound_sockets=127.7.194.194
--tserver_master_addrs=127.7.194.254:45193
--webserver_port=36081
--webserver_interface=127.7.194.194
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab
--principal=oryx/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.194.254:45193,127.7.194.253:35843
--tserver_master_addrs=127.7.194.254:45193,127.7.194.253:35843,127.7.194.252:37941 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:34:57.282155 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.264298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.193:34451)
0619 02:34:57.265191 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.265246 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.266341 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.266970 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.266978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.267476 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.268039 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.268052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.268866 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.268870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.269843 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.269852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.270368 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.270372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.270549 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.274151 (+ 3602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.274163 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.275495 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.278066 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.278069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.278071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.278538 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.279160 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.279163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.279165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.279252 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.279707 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.279710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.279796 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.280444 (+ 648us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.280929 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":67,"thread_start_us":553,"threads_started":1}
I20260619 02:34:57.282178 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.266110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.7.194.193:36383)
0619 02:34:57.266397 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.266409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.266489 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.266973 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.266980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.267521 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.268051 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.268060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.269141 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.269144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.270085 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.270094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.270819 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.270823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.270919 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.272118 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.272135 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.275082 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.277426 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.277435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.277459 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.277871 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.278299 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.278305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.278307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.278531 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.279205 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.279214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.279577 (+ 363us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.280437 (+ 860us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.280976 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":293,"spinlock_wait_cycles":37376,"thread_start_us":66,"threads_started":1}
I20260619 02:34:57.283043 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.265790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:37941 (local address 127.7.194.193:60913)
0619 02:34:57.266150 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.266164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.266346 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.266973 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.266980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.267516 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.268050 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.268060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.269354 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.269357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.270237 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.270244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.270934 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.270937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.271058 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.272273 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.272290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.275082 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.278147 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.278153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.278158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.278647 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.278982 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.278987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.278992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.279086 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.279612 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.279616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.279729 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.281155 (+ 1426us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.281333 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":97,"spinlock_wait_cycles":23296,"thread_start_us":83,"threads_started":1}
I20260619 02:34:57.283450 15043 heartbeater.cc:347] Connected to a master server at 127.7.194.254:45193
I20260619 02:34:57.283557 15044 heartbeater.cc:347] Connected to a master server at 127.7.194.253:35843
I20260619 02:34:57.283710 15044 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:57.284128 15042 heartbeater.cc:347] Connected to a master server at 127.7.194.252:37941
I20260619 02:34:57.284206 15042 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:57.284354 15042 heartbeater.cc:511] Master 127.7.194.252:37941 requested a full tablet report, sending...
I20260619 02:34:57.283710 15043 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:57.284354 15044 heartbeater.cc:511] Master 127.7.194.253:35843 requested a full tablet report, sending...
I20260619 02:34:57.284668 15043 heartbeater.cc:511] Master 127.7.194.254:45193 requested a full tablet report, sending...
I20260619 02:34:57.285979 14072 ts_manager.cc:194] Re-registered known tserver with Master: baa5e2361000406e892e89dac775c4b7 (127.7.194.193:42707)
I20260619 02:34:57.286077 14673 ts_manager.cc:194] Registered new tserver with Master: baa5e2361000406e892e89dac775c4b7 (127.7.194.193:42707)
I20260619 02:34:57.286185 14855 ts_manager.cc:194] Registered new tserver with Master: baa5e2361000406e892e89dac775c4b7 (127.7.194.193:42707)
I20260619 02:34:57.286778 14072 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:34451
W20260619 02:34:57.398635 15050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:57.398905 15050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:57.398952 15050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:57.403064 15050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:57.403129 15050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:57.403190 15050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:34:57.408416 15050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab
--principal=oryx/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:40645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=36081
--tserver_master_addrs=127.7.194.254:45193,127.7.194.253:35843,127.7.194.252:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:57.409513 15050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:57.410657 15050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:34:57.412089 14531 heartbeater.cc:503] Master 127.7.194.254:45193 was elected leader, sending a full tablet report...
W20260619 02:34:57.420768 15058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:57.420781 15055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:57.420790 15056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:57.421022 15050 server_base.cc:1034] running on GCE node
I20260619 02:34:57.421509 15050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:57.422191 15050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:57.423385 15050 hybrid_clock.cc:648] HybridClock initialized: now 1781836497423340 us; error 57 us; skew 500 ppm
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:57.426340 15050 init.cc:377] Logged in from keytab as oryx/127.7.194.194@KRBTEST.COM (short username oryx)
I20260619 02:34:57.427203 15050 webserver.cc:460] Webserver started at http://127.7.194.194:36081/ using document root <none> and password file <none>
I20260619 02:34:57.427733 15050 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:57.427802 15050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:57.431900 15050 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:34:57.434244 15065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.435546 15050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:34:57.435675 15050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "b090dd45ea8e4f3294be929815ed3472"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "e69241e92db2d9a3bad642a19700b0e0"
server_key_iv: "c594fb35a76d00777b3f989927f80716"
server_key_version: "encryptionkey@0"
I20260619 02:34:57.436298 15050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:57.449314 15050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:57.452638 15050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:57.452790 15050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:57.453399 15050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:57.454320 15050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:57.454366 15050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.454434 15050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:57.454464 15050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.463502 15050 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:40645
I20260619 02:34:57.463569 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:40645 every 8 connection(s)
I20260619 02:34:57.464655 15050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:34:57.467725 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 15050
I20260619 02:34:57.467950 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 14402
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.253@KRBTEST.COM
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.252@KRBTEST.COM
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:34:57.477746 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:40641
--local_ip_for_outbound_sockets=127.7.194.195
--tserver_master_addrs=127.7.194.254:45193
--webserver_port=41411
--webserver_interface=127.7.194.195
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab
--principal=oryx/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.194.254:45193,127.7.194.253:35843
--tserver_master_addrs=127.7.194.254:45193,127.7.194.253:35843,127.7.194.252:37941 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:34:57.483304 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.469196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:59323 (local address 127.7.194.253:35843)
0619 02:34:57.469325 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.469332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.469348 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:34:57.469409 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.469414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.469469 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.469601 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.471112 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.471642 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.473265 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.473637 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.473709 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.478446 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.478453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.478459 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.478498 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.480711 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.481344 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.481347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.481349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.481436 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.481828 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.481832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.481833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.482087 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.482158 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.482980 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.483109 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:34:57.483393 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.469196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:49601 (local address 127.7.194.254:45193)
0619 02:34:57.472263 (+ 3067us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.472268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.472281 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:34:57.472322 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.472325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.472372 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.472458 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.472944 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.473738 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.474833 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.475205 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.475293 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.478209 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.478214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.478217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.478247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.480436 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.481255 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.481261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.481266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.481358 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.481835 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.481841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.481845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.482116 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.482196 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.483004 (+ 808us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.483152 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2964}
I20260619 02:34:57.484375 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.468372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.7.194.194:59323)
0619 02:34:57.468674 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.468689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.469013 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.469786 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.469793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.470413 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.470966 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.470979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.471788 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.471791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.473133 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.473142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.473909 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.473912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.474102 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.474952 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.474966 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.478278 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.480828 (+ 2550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.480834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.480836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.481223 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.481569 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.481571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.481574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.481714 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.482236 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.482238 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.482676 (+ 438us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.483450 (+ 774us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.483575 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":123,"spinlock_wait_cycles":33024,"thread_start_us":82,"threads_started":1}
I20260619 02:34:57.484660 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.467351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:37941 (local address 127.7.194.194:40159)
0619 02:34:57.467824 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.467877 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.469369 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.469788 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.469796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.470419 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.470969 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.470984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.472201 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.472204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.473142 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.473155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.474023 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.474029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.474201 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.474876 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.474902 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.478067 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.480750 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.480755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.480758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.481223 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.481564 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.481567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.481569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.481714 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.482221 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.482229 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.482679 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.482913 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.483267 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":99,"spinlock_wait_cycles":32000,"thread_start_us":142,"threads_started":1}
I20260619 02:34:57.485123 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.466559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:40159 (local address 127.7.194.252:37941)
0619 02:34:57.466744 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.466751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.467991 (+ 1240us) server_negotiation.cc:407] Connection header received
0619 02:34:57.469456 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.469462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.469529 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.469629 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.471124 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.472039 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.473289 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.473762 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.473856 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.478241 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.478248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.478255 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.478291 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.480609 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.481344 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.481347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.481349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.481436 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.481829 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.481832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.481834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.482078 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.482165 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.484817 (+ 2652us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.484939 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:34:57.485952 15181 heartbeater.cc:347] Connected to a master server at 127.7.194.253:35843
I20260619 02:34:57.486056 15179 heartbeater.cc:347] Connected to a master server at 127.7.194.252:37941
I20260619 02:34:57.485961 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.468733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.194:49601)
0619 02:34:57.468945 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.468959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.469033 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.472491 (+ 3458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.472495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.472687 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.472798 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.472804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.473860 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.473863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.474698 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.474710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.475449 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.475452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.475551 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.476187 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.476207 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.478017 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.480582 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.480588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.480609 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.481138 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.481472 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.481477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.481482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.481721 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.482274 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.482279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.482680 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.484079 (+ 1399us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.484264 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":62,"spinlock_wait_cycles":15360,"thread_start_us":76,"threads_started":1}
I20260619 02:34:57.486245 15181 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:57.486245 15179 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:57.486907 15180 heartbeater.cc:347] Connected to a master server at 127.7.194.254:45193
I20260619 02:34:57.486939 15181 heartbeater.cc:511] Master 127.7.194.253:35843 requested a full tablet report, sending...
I20260619 02:34:57.486992 15180 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:57.486943 15179 heartbeater.cc:511] Master 127.7.194.252:37941 requested a full tablet report, sending...
I20260619 02:34:57.487124 15180 heartbeater.cc:511] Master 127.7.194.254:45193 requested a full tablet report, sending...
I20260619 02:34:57.488097 14072 ts_manager.cc:194] Re-registered known tserver with Master: b090dd45ea8e4f3294be929815ed3472 (127.7.194.194:40645)
I20260619 02:34:57.488250 14673 ts_manager.cc:194] Registered new tserver with Master: b090dd45ea8e4f3294be929815ed3472 (127.7.194.194:40645)
I20260619 02:34:57.488265 14855 ts_manager.cc:194] Registered new tserver with Master: b090dd45ea8e4f3294be929815ed3472 (127.7.194.194:40645)
I20260619 02:34:57.488826 14072 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:49601
I20260619 02:34:57.580444 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.576544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:37941 (local address 127.0.0.1:36550)
0619 02:34:57.576689 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.576707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.576808 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.577150 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.577152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.577168 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:57.577256 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.577261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.578381 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.578384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.579147 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.579158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.580106 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.580109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.580181 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.580202 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.580257 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.580323 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:34:57.580495 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.576633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.7.194.252:37941)
0619 02:34:57.576782 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.576788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.576807 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:34:57.576923 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.576928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.577002 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.577123 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:57.577364 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.578234 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.579287 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.579966 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.580089 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.580252 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.580288 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.580362 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:34:57.581212 14865 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 02:34:57.581523 14905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ef3db455d5435da039949bfaa1ac86. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.581758 14905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: This master's current role is: LEARNER
W20260619 02:34:57.602025 15186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:34:57.602305 15186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:34:57.602349 15186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:34:57.606496 15186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:34:57.606565 15186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:34:57.606633 15186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:34:57.612344 15186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab
--principal=oryx/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:40641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=41411
--tserver_master_addrs=127.7.194.254:45193,127.7.194.253:35843,127.7.194.252:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:34:57.613462 15186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:34:57.614739 15186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:34:57.624801 15194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:34:57.624843 15192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:57.625116 15186 server_base.cc:1034] running on GCE node
W20260619 02:34:57.624801 15191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:34:57.625713 15186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:34:57.626397 15186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:34:57.627580 15186 hybrid_clock.cc:648] HybridClock initialized: now 1781836497627545 us; error 45 us; skew 500 ppm
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:34:57.630475 15186 init.cc:377] Logged in from keytab as oryx/127.7.194.195@KRBTEST.COM (short username oryx)
I20260619 02:34:57.631472 15186 webserver.cc:460] Webserver started at http://127.7.194.195:41411/ using document root <none> and password file <none>
I20260619 02:34:57.632110 15186 fs_manager.cc:359] Metadata directory not provided
I20260619 02:34:57.632194 15186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:34:57.636318 15186 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:34:57.638552 15201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.639947 15186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:34:57.640072 15186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "4ff80a9a53a84a5f9a8b60a334680d98"
format_stamp: "Formatted at 2026-06-19 02:34:56 on dist-test-slave-9tkz"
server_key: "f05b3ab4750cb6cf4128af2a04d6e85b"
server_key_iv: "913c5bf3fafa909308eac02a35eb5913"
server_key_version: "encryptionkey@0"
I20260619 02:34:57.640676 15186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:34:57.654249 15186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:34:57.657780 15186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:34:57.657989 15186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:34:57.658636 15186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:34:57.659596 15186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:34:57.659646 15186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.659708 15186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:34:57.659739 15186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:34:57.669685 15186 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:40641
I20260619 02:34:57.669720 15314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:40641 every 8 connection(s)
I20260619 02:34:57.670739 15186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:34:57.675758 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 15186
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.252@KRBTEST.COM
Jun 19 02:34:57 dist-test-slave-9tkz krb5kdc[14027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836497, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.253@KRBTEST.COM
I20260619 02:34:57.689236 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.672633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:33895 (local address 127.7.194.253:35843)
0619 02:34:57.672821 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.672825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.674158 (+ 1333us) server_negotiation.cc:407] Connection header received
0619 02:34:57.675424 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.675428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.675482 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.675535 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.677657 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.678430 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.682459 (+ 4029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.682814 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.682890 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.685681 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.685686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.685689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.685721 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.687484 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.687940 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.687944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.687947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.688033 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.688358 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.688361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.688363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.688579 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.688653 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.688962 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.689080 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 02:34:57.689536 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.675116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:40417 (local address 127.7.194.252:37941)
0619 02:34:57.675244 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.675248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.675262 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:57.675356 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.675358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.675429 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.675504 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.677443 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.678243 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.680653 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.681043 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.681132 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.685983 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.685989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.685993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.686023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.687589 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.688099 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.688102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.688104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.688186 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.688686 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.688691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.688696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.688901 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.688976 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.689294 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.689404 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:34:57.689539 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.679006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.7.194.254:45193)
0619 02:34:57.679238 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.679244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.679257 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:34:57.679296 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.679298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.679343 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.680376 (+ 1033us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.680463 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.681200 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.682462 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.682803 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.682901 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.684505 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.684509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.684511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.684537 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.687502 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.688007 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.688012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.688017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.688102 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.688426 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.688429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.688430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.688644 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.688765 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.689246 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.689366 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260619 02:34:57.690169 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.674825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:37941 (local address 127.7.194.195:40417)
0619 02:34:57.675045 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.675058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.675163 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.675825 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.675834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.676490 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.677269 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.677286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.678437 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.678443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.680526 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.680536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.681297 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.681301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.681459 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.681979 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.681994 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.685105 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.687740 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.687742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.687744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.687993 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.688431 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.688433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.688436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.688514 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.689049 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.689051 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.689126 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.689889 (+ 763us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.690033 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1364,"spinlock_wait_cycles":6272,"thread_start_us":76,"threads_started":1}
I20260619 02:34:57.690395 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.674451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.195:42751)
0619 02:34:57.674813 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.674826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.675173 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.675823 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.675833 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.676623 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.677267 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.677283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.678474 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.678479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.680045 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.680057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.680779 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.680782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.680978 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.681666 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.681688 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.684180 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.686535 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.686541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.686557 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.686959 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.687292 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.687295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.687297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.687481 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.687973 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.687981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.688322 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.688536 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.688920 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":138,"spinlock_wait_cycles":13184,"thread_start_us":98,"threads_started":1}
I20260619 02:34:57.690511 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.673537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.7.194.195:33895)
0619 02:34:57.674002 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.674046 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.675171 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.675875 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.675878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.676488 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:34:57.677478 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.677485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.678554 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.678556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.681407 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.681416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.683051 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.683055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.683150 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.683766 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:34:57.683777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:34:57.685481 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:34:57.687595 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.687598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.687600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.687824 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.688152 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:34:57.688154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:34:57.688156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:34:57.688254 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:34:57.688706 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:34:57.688709 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:34:57.688785 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.690096 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.690255 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":101,"thread_start_us":111,"threads_started":1}
I20260619 02:34:57.690884 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.674596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:42751 (local address 127.7.194.254:45193)
0619 02:34:57.674769 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.674774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.674913 (+ 139us) server_negotiation.cc:407] Connection header received
0619 02:34:57.675425 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.675428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.675482 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.675535 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.677420 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.678322 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.680190 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.680519 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.680620 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.684352 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.684359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.684363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.684390 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.686391 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.687086 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.687092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.687094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.687181 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.687607 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.687611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.687613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.687832 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.687923 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.690585 (+ 2662us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.690699 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":68}
I20260619 02:34:57.692492 15315 heartbeater.cc:347] Connected to a master server at 127.7.194.252:37941
I20260619 02:34:57.692598 15317 heartbeater.cc:347] Connected to a master server at 127.7.194.253:35843
I20260619 02:34:57.692781 15315 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:57.692780 15317 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:57.692493 15316 heartbeater.cc:347] Connected to a master server at 127.7.194.254:45193
I20260619 02:34:57.693308 15316 heartbeater.cc:464] Registering TS with master...
I20260619 02:34:57.693396 15317 heartbeater.cc:511] Master 127.7.194.253:35843 requested a full tablet report, sending...
I20260619 02:34:57.693395 15315 heartbeater.cc:511] Master 127.7.194.252:37941 requested a full tablet report, sending...
I20260619 02:34:57.693466 15316 heartbeater.cc:511] Master 127.7.194.254:45193 requested a full tablet report, sending...
I20260619 02:34:57.694547 14072 ts_manager.cc:194] Re-registered known tserver with Master: 4ff80a9a53a84a5f9a8b60a334680d98 (127.7.194.195:40641)
I20260619 02:34:57.694820 14855 ts_manager.cc:194] Registered new tserver with Master: 4ff80a9a53a84a5f9a8b60a334680d98 (127.7.194.195:40641)
I20260619 02:34:57.695312 14072 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:42751
I20260619 02:34:57.695453 14673 ts_manager.cc:194] Registered new tserver with Master: 4ff80a9a53a84a5f9a8b60a334680d98 (127.7.194.195:40641)
I20260619 02:34:57.698961 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.679414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.7.194.252:37941)
0619 02:34:57.680007 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.680018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.680297 (+ 279us) server_negotiation.cc:407] Connection header received
0619 02:34:57.680334 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.680337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.680378 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.680449 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.681962 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.684673 (+ 2711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.691486 (+ 6813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.691877 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.691989 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.695172 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.695175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.695178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.695202 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.697033 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.697499 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.697502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.697504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.697610 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.697918 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.697925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.697928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.698156 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.698293 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.698656 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.698781 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":71,"threads_started":1}
I20260619 02:34:57.696234 14072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:34:57.700660 14072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:57.702605 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.679216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.7.194.253:35843)
0619 02:34:57.683455 (+ 4239us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.683461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.683475 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:57.683516 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.683520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.683581 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.691401 (+ 7820us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.691502 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.692267 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.695817 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.697520 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.697682 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.699335 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.699338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.699341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.699371 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.700971 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.701373 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.701378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.701380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.701458 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.701744 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.701747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.701749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.701918 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.702055 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.702287 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.702432 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4151,"thread_start_us":78,"threads_started":1}
I20260619 02:34:57.703392 14826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: NON_VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: true } } }
I20260619 02:34:57.703522 14826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:57.705339 14826 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: attempting to promote NON_VOTER 91e26bb975234ecaadc52946641ecd68 to VOTER
I20260619 02:34:57.705751 14826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } }
I20260619 02:34:57.706881 14865 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 LEARNER]: Refusing update from remote peer 71ef3db455d5435da039949bfaa1ac86: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 02:34:57.706983 14683 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Refusing update from remote peer 71ef3db455d5435da039949bfaa1ac86: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 02:34:57.707227 15334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 02:34:57.707367 15333 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 02:34:57.708922 15334 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 91e26bb975234ecaadc52946641ecd68 (127.7.194.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:57.709357 14865 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 91e26bb975234ecaadc52946641ecd68 (127.7.194.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:57.709393 14683 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 91e26bb975234ecaadc52946641ecd68 (127.7.194.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:57.710556 15334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:57.710682 15334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:57.710815 15334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:57.710901 15334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:57.711000 15334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:57.711079 15334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86 [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:57.711055 14905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:57.711181 14905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:34:57.711798 15342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ef3db455d5435da039949bfaa1ac86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:57.711915 15342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:34:57.723743 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.717360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:40641 (local address 127.0.0.1:37050)
0619 02:34:57.717485 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.717501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.717621 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.718652 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.718656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.718676 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:57.718810 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.718817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.720877 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.720883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.722096 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.722106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.723365 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.723369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.723454 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.723477 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.723565 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.723635 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:34:57.723809 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.717715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37050 (local address 127.7.194.195:40641)
0619 02:34:57.718088 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.718098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.718118 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:34:57.718184 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.718189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.718426 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.718986 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:57.719140 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.720768 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.722238 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.723226 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.723353 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.723495 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.723602 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.723684 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":176,"threads_started":1}
I20260619 02:34:57.724308 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.718499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:42707 (local address 127.0.0.1:42188)
0619 02:34:57.718761 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.718777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.718870 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.719593 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.719597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.719612 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:57.719704 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.719712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.721116 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.721119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.722254 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.722262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.723950 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.723955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.724043 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.724065 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.724148 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.724216 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
I20260619 02:34:57.724403 15345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.718650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.7.194.193:42707)
0619 02:34:57.719101 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.719115 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.719159 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:34:57.719240 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.719245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.719464 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.719628 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:57.719819 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.721001 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.722377 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.723233 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.723359 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.723497 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.724182 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.724277 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":110,"threads_started":1}
I20260619 02:34:57.724943 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.718847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:40645 (local address 127.0.0.1:47246)
0619 02:34:57.719064 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.719077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.719153 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.720071 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.720074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.720093 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:57.720192 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.720198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.721383 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.721390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.722339 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.722351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.724556 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.724559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.724643 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.724671 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.724762 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.724829 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260619 02:34:57.725987 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.719252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.7.194.194:40645)
0619 02:34:57.719597 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.719609 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.719641 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:34:57.719712 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.719722 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.719924 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.720071 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:57.720292 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.721279 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.722451 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.723226 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.723343 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.723495 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.725744 (+ 2249us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.725852 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
I20260619 02:34:57.726994 14976 tablet_service.cc:1467] Processing CreateTablet for tablet db9cde15f6d644c1a6c63d5d0772f0c4 (DEFAULT_TABLE table=test-table [id=50d1471b9c454fd8b41487a38ddbf317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:57.727035 15249 tablet_service.cc:1467] Processing CreateTablet for tablet db9cde15f6d644c1a6c63d5d0772f0c4 (DEFAULT_TABLE table=test-table [id=50d1471b9c454fd8b41487a38ddbf317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:57.727967 14976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9cde15f6d644c1a6c63d5d0772f0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:57.727969 15249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9cde15f6d644c1a6c63d5d0772f0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:57.734077 15113 tablet_service.cc:1467] Processing CreateTablet for tablet db9cde15f6d644c1a6c63d5d0772f0c4 (DEFAULT_TABLE table=test-table [id=50d1471b9c454fd8b41487a38ddbf317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:57.735253 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9cde15f6d644c1a6c63d5d0772f0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:57.735392 15348 tablet_bootstrap.cc:492] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: Bootstrap starting.
I20260619 02:34:57.736776 15349 tablet_bootstrap.cc:492] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: Bootstrap starting.
I20260619 02:34:57.737789 15348 tablet_bootstrap.cc:654] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:57.738529 15348 log.cc:824] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:57.739674 15349 tablet_bootstrap.cc:654] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:57.740392 15348 tablet_bootstrap.cc:492] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: No bootstrap required, opened a new log
I20260619 02:34:57.740525 15349 log.cc:824] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:57.740613 15348 ts_tablet_manager.cc:1397] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:34:57.742578 15349 tablet_bootstrap.cc:492] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: No bootstrap required, opened a new log
I20260619 02:34:57.742770 15349 ts_tablet_manager.cc:1397] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:34:57.743507 15352 tablet_bootstrap.cc:492] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: Bootstrap starting.
I20260619 02:34:57.744817 15348 raft_consensus.cc:348] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } }
I20260619 02:34:57.745121 15348 raft_consensus.cc:374] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:57.745244 15348 raft_consensus.cc:729] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5e2361000406e892e89dac775c4b7, State: Initialized, Role: FOLLOWER
I20260619 02:34:57.745738 15348 consensus_queue.cc:260] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } }
I20260619 02:34:57.746496 15352 tablet_bootstrap.cc:654] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:57.747200 15348 ts_tablet_manager.cc:1428] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260619 02:34:57.747331 15349 raft_consensus.cc:348] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } }
I20260619 02:34:57.747584 15349 raft_consensus.cc:374] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:57.747605 15043 heartbeater.cc:503] Master 127.7.194.254:45193 was elected leader, sending a full tablet report...
I20260619 02:34:57.747658 15349 raft_consensus.cc:729] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff80a9a53a84a5f9a8b60a334680d98, State: Initialized, Role: FOLLOWER
I20260619 02:34:57.747840 15352 log.cc:824] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: Log is configured to *not* fsync() on all Append() calls
I20260619 02:34:57.748065 15349 consensus_queue.cc:260] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } }
I20260619 02:34:57.749012 15349 ts_tablet_manager.cc:1428] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:34:57.750710 15316 heartbeater.cc:503] Master 127.7.194.254:45193 was elected leader, sending a full tablet report...
I20260619 02:34:57.751469 15352 tablet_bootstrap.cc:492] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: No bootstrap required, opened a new log
I20260619 02:34:57.752779 15352 ts_tablet_manager.cc:1397] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:34:57.755976 15352 raft_consensus.cc:348] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } }
I20260619 02:34:57.756222 15352 raft_consensus.cc:374] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:57.756278 15352 raft_consensus.cc:729] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b090dd45ea8e4f3294be929815ed3472, State: Initialized, Role: FOLLOWER
I20260619 02:34:57.756685 15352 consensus_queue.cc:260] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } }
I20260619 02:34:57.757375 15180 heartbeater.cc:503] Master 127.7.194.254:45193 was elected leader, sending a full tablet report...
I20260619 02:34:57.757676 15352 ts_tablet_manager.cc:1428] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 02:34:57.764014 15045 tablet.cc:2369] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:34:57.779078 15355 raft_consensus.cc:482] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:57.779274 15355 raft_consensus.cc:504] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } }
I20260619 02:34:57.780388 15355 leader_election.cc:288] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa5e2361000406e892e89dac775c4b7 (127.7.194.193:42707), b090dd45ea8e4f3294be929815ed3472 (127.7.194.194:40645)
I20260619 02:34:57.784287 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.781069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:43319 (local address 127.7.194.194:40645)
0619 02:34:57.781202 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.781206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.781220 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:57.781268 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.781271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.781314 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.781398 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:57.781667 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.782139 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.783206 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.783729 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.783852 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.783911 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.784070 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.784150 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:34:57.784291 15345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.780772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:41513 (local address 127.7.194.193:42707)
0619 02:34:57.780915 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.780922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.780938 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:34:57.781058 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.781061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.781120 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.781203 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:57.781482 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.782097 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.783303 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.783742 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.783875 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.783932 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.784110 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.784181 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:34:57.784296 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.780657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:42707 (local address 127.7.194.195:41513)
0619 02:34:57.780799 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.780817 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.780942 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.781215 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.781219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.781239 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:57.781374 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.781383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.782203 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.782208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.783161 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.783174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.783870 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.783873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.783979 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.783996 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.784086 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.784156 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260619 02:34:57.784303 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.780939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:40645 (local address 127.7.194.195:43319)
0619 02:34:57.781080 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:57.781093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:57.781180 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:57.781412 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:57.781414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:57.781427 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:57.781548 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.781571 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.782270 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.782272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.783078 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:57.783087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.783841 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.783844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:57.783942 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.783970 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:34:57.784036 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:34:57.784137 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 02:34:57.785110 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9cde15f6d644c1a6c63d5d0772f0c4" candidate_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e2361000406e892e89dac775c4b7" is_pre_election: true
I20260619 02:34:57.785487 14996 raft_consensus.cc:2454] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff80a9a53a84a5f9a8b60a334680d98 in term 0.
I20260619 02:34:57.785516 15133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9cde15f6d644c1a6c63d5d0772f0c4" candidate_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b090dd45ea8e4f3294be929815ed3472" is_pre_election: true
I20260619 02:34:57.785840 15133 raft_consensus.cc:2454] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff80a9a53a84a5f9a8b60a334680d98 in term 0.
I20260619 02:34:57.786017 15203 leader_election.cc:302] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff80a9a53a84a5f9a8b60a334680d98, baa5e2361000406e892e89dac775c4b7; no voters:
I20260619 02:34:57.786343 15355 raft_consensus.cc:2781] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:57.786485 15355 raft_consensus.cc:482] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:57.786542 15355 raft_consensus.cc:3037] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:57.787653 15355 raft_consensus.cc:504] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } }
I20260619 02:34:57.787992 15355 leader_election.cc:288] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [CANDIDATE]: Term 1 election: Requested vote from peers baa5e2361000406e892e89dac775c4b7 (127.7.194.193:42707), b090dd45ea8e4f3294be929815ed3472 (127.7.194.194:40645)
I20260619 02:34:57.788367 15133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9cde15f6d644c1a6c63d5d0772f0c4" candidate_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b090dd45ea8e4f3294be929815ed3472"
I20260619 02:34:57.788367 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9cde15f6d644c1a6c63d5d0772f0c4" candidate_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e2361000406e892e89dac775c4b7"
I20260619 02:34:57.788494 15133 raft_consensus.cc:3037] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:57.788494 14996 raft_consensus.cc:3037] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:57.789815 15133 raft_consensus.cc:2454] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff80a9a53a84a5f9a8b60a334680d98 in term 1.
I20260619 02:34:57.789815 14996 raft_consensus.cc:2454] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff80a9a53a84a5f9a8b60a334680d98 in term 1.
I20260619 02:34:57.790076 15203 leader_election.cc:302] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff80a9a53a84a5f9a8b60a334680d98, b090dd45ea8e4f3294be929815ed3472; no voters:
I20260619 02:34:57.790256 15355 raft_consensus.cc:2781] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:57.790478 15355 raft_consensus.cc:686] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 1 LEADER]: Becoming Leader. State: Replica: 4ff80a9a53a84a5f9a8b60a334680d98, State: Running, Role: LEADER
I20260619 02:34:57.790848 15355 consensus_queue.cc:237] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } }
I20260619 02:34:57.794441 14072 catalog_manager.cc:5495] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ff80a9a53a84a5f9a8b60a334680d98 (127.7.194.195). New cstate: current_term: 1 leader_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } health_report { overall_health: UNKNOWN } } }
I20260619 02:34:57.820297 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.816422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.7.194.195:40641)
0619 02:34:57.816570 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.816576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.816592 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:34:57.816675 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.816677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.816734 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.816844 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:57.817074 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.817676 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.818897 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.819101 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.819180 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.819477 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:57.819910 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:57.819999 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.820092 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.820144 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:34:57.826682 15133 raft_consensus.cc:1261] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [term 1 FOLLOWER]: Refusing update from remote peer 4ff80a9a53a84a5f9a8b60a334680d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:57.826719 14996 raft_consensus.cc:1261] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [term 1 FOLLOWER]: Refusing update from remote peer 4ff80a9a53a84a5f9a8b60a334680d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:57.827389 15355 consensus_queue.cc:1035] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:57.827603 15357 consensus_queue.cc:1035] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:57.837275 15361 mvcc.cc:205] Tried to move back new op lower bound from 7298402295090864128 to 7298402294954426368. Current Snapshot: MvccSnapshot[applied={T|T < 7298402295090864128}]
I20260619 02:34:57.844630 14072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:57.844802 14072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:57.848583 14072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71ef3db455d5435da039949bfaa1ac86: Sending DeleteTablet for 3 replicas of tablet db9cde15f6d644c1a6c63d5d0772f0c4
I20260619 02:34:57.849705 14976 tablet_service.cc:1514] Processing DeleteTablet for tablet db9cde15f6d644c1a6c63d5d0772f0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:57 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:42188
I20260619 02:34:57.849872 15249 tablet_service.cc:1514] Processing DeleteTablet for tablet db9cde15f6d644c1a6c63d5d0772f0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:57 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:37050
I20260619 02:34:57.849870 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet db9cde15f6d644c1a6c63d5d0772f0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:57 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:47246
I20260619 02:34:57.850358 15372 tablet_replica.cc:330] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: stopping tablet replica
I20260619 02:34:57.850694 15372 raft_consensus.cc:2229] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:57.851037 15372 raft_consensus.cc:2258] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:57.852720 15372 ts_tablet_manager.cc:1905] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:57.854316 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 14040
I20260619 02:34:57.858218 15372 ts_tablet_manager.cc:1918] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:57.858331 15372 log.cc:1196] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/db9cde15f6d644c1a6c63d5d0772f0c4
I20260619 02:34:57.858672 15372 ts_tablet_manager.cc:1939] T db9cde15f6d644c1a6c63d5d0772f0c4 P b090dd45ea8e4f3294be929815ed3472: Deleting consensus metadata
I20260619 02:34:57.861799 15371 tablet_replica.cc:330] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: stopping tablet replica
I20260619 02:34:57.862227 15371 raft_consensus.cc:2229] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 1 LEADER]: Raft consensus shutting down.
W20260619 02:34:57.862334 15068 connection.cc:535] server connection from 127.0.0.1:47246 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:34:57.862702 15371 raft_consensus.cc:2258] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:57.863268 15370 tablet_replica.cc:330] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: stopping tablet replica
I20260619 02:34:57.864387 15371 ts_tablet_manager.cc:1905] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:57.866719 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.851845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.7.194.253:35843)
0619 02:34:57.851963 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.851967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.855051 (+ 3084us) server_negotiation.cc:407] Connection header received
0619 02:34:57.855263 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.855269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.855333 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.855389 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.856511 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.857339 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.858816 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.859171 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.859245 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.861568 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.861575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.861581 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.861612 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.864293 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.864888 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.864892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.864894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.864991 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.865297 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.865301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.865303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.865513 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.865665 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.866383 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.866522 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:34:57.867280 15370 raft_consensus.cc:2229] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:57.867679 15370 raft_consensus.cc:2258] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:57.868500 15371 ts_tablet_manager.cc:1918] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:57.870077 15370 ts_tablet_manager.cc:1905] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:57.870234 15371 log.cc:1196] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/db9cde15f6d644c1a6c63d5d0772f0c4
I20260619 02:34:57.870643 15371 ts_tablet_manager.cc:1939] T db9cde15f6d644c1a6c63d5d0772f0c4 P 4ff80a9a53a84a5f9a8b60a334680d98: Deleting consensus metadata
W20260619 02:34:57.871738 15203 connection.cc:441] server connection from 127.0.0.1:37050 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37050 (request call id 1) could send its response
I20260619 02:34:57.874704 15370 ts_tablet_manager.cc:1918] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:57.874890 15370 log.cc:1196] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/db9cde15f6d644c1a6c63d5d0772f0c4
I20260619 02:34:57.875286 15370 ts_tablet_manager.cc:1939] T db9cde15f6d644c1a6c63d5d0772f0c4 P baa5e2361000406e892e89dac775c4b7: Deleting consensus metadata
W20260619 02:34:57.876510 14929 connection.cc:441] server connection from 127.0.0.1:42188 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42188 (request call id 1) could send its response
I20260619 02:34:57.877506 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.851897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.7.194.252:37941)
0619 02:34:57.852007 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.852012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.867862 (+ 15850us) server_negotiation.cc:407] Connection header received
0619 02:34:57.868056 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.868060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.868133 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.868195 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.869135 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.869987 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.871149 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.871488 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.871568 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.873423 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.873427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.873430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.873454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.875224 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.875690 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.875694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.875695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.875777 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.876336 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.876339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.876341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.876627 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.876738 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.877198 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.877335 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":40}
I20260619 02:34:57.882154 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.866758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.7.194.253:35843)
0619 02:34:57.866876 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.866882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.868177 (+ 1295us) server_negotiation.cc:407] Connection header received
0619 02:34:57.868564 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.868569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.868639 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.868696 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.870809 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.871533 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.875017 (+ 3484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.875396 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.875498 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.878128 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.878132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.878135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.878165 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.879967 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.880450 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.880456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.880458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.880545 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.880869 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.880873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.880874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.881121 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.881250 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.881466 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.881973 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:34:57.882292 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:57.867279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.7.194.252:37941)
0619 02:34:57.867391 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:57.867395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:57.868489 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 02:34:57.869758 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:57.869761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:57.869803 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:57.869848 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:34:57.872776 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.873706 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.876025 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:57.876366 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:57.876436 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:57.879030 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:34:57.879037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:34:57.879039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:34:57.879067 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:34:57.880493 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.880977 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.880980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.880982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.881052 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:34:57.881332 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:34:57.881337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:34:57.881338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:34:57.881527 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:34:57.881721 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:57.882029 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 02:34:57.882138 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":43}
I20260619 02:34:58.839649 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:58.839069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.193:46949)
0619 02:34:58.839335 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:58.839497 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
W20260619 02:34:58.839982 15043 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:45193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
I20260619 02:34:58.843304 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:58.842768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.195:43361)
0619 02:34:58.843095 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:58.843193 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":90,"threads_started":1}
W20260619 02:34:58.843762 15316 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:45193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
I20260619 02:34:58.843842 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:58.843192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.194:45569)
0619 02:34:58.843614 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:58.843702 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":69,"threads_started":1}
W20260619 02:34:58.844107 15180 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:45193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
I20260619 02:34:59.350364 15384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71ef3db455d5435da039949bfaa1ac86)
I20260619 02:34:59.350566 15384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } }
I20260619 02:34:59.351526 15384 leader_election.cc:288] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71ef3db455d5435da039949bfaa1ac86 (127.7.194.254:45193), 91e26bb975234ecaadc52946641ecd68 (127.7.194.252:37941)
I20260619 02:34:59.352208 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.351734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:40364)
0619 02:34:59.351980 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.352072 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
W20260619 02:34:59.352519 14659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
I20260619 02:34:59.353468 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.353219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:40370)
0619 02:34:59.353333 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.353400 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:34:59.353948 15389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71ef3db455d5435da039949bfaa1ac86)
I20260619 02:34:59.354072 15389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } }
W20260619 02:34:59.353662 14659 leader_election.cc:334] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71ef3db455d5435da039949bfaa1ac86 (127.7.194.254:45193): Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
I20260619 02:34:59.354802 15389 leader_election.cc:288] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71ef3db455d5435da039949bfaa1ac86 (127.7.194.254:45193), 905b3b91feb74d8ca2904dea1064bedd (127.7.194.253:35843)
I20260619 02:34:59.355266 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.354902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:40376)
0619 02:34:59.355084 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.355163 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
W20260619 02:34:59.355491 14849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
I20260619 02:34:59.356609 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.356378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:40384)
0619 02:34:59.356489 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.356544 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:34:59.356734 14849 leader_election.cc:334] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71ef3db455d5435da039949bfaa1ac86 (127.7.194.254:45193): Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
I20260619 02:34:59.356832 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.352319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.7.194.252:37941)
0619 02:34:59.352564 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:59.352569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:59.352582 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:34:59.352651 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:59.352655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:59.352708 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:59.352790 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:59.353072 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.353939 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.355469 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.355979 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.356049 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.356112 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:59.356494 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 02:34:59.356570 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260619 02:34:59.356866 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.352109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:37941 (local address 127.0.0.1:33192)
0619 02:34:59.352407 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.352422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:59.352532 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:59.352814 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:59.352816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:59.352838 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:59.352951 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.352956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.354110 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.354114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.355308 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.355321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.356206 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.356209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.356338 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.356377 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 02:34:59.356629 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 02:34:59.356743 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260619 02:34:59.357338 14865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "905b3b91feb74d8ca2904dea1064bedd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91e26bb975234ecaadc52946641ecd68" is_pre_election: true
I20260619 02:34:59.357573 14865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905b3b91feb74d8ca2904dea1064bedd in term 1.
I20260619 02:34:59.357926 14659 leader_election.cc:302] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 905b3b91feb74d8ca2904dea1064bedd, 91e26bb975234ecaadc52946641ecd68; no voters: 71ef3db455d5435da039949bfaa1ac86
I20260619 02:34:59.358170 15384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:34:59.358242 15384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71ef3db455d5435da039949bfaa1ac86)
I20260619 02:34:59.358281 15384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:34:59.359244 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.355188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:35843 (local address 127.0.0.1:38602)
0619 02:34:59.355440 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.355454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:59.355539 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:59.355979 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:59.355983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:59.356006 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:59.356088 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.356093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.357041 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.357047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.358129 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.358139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.358850 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.358854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.358949 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.358972 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:34:59.359045 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:34:59.359113 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":54,"threads_started":1}
I20260619 02:34:59.359225 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.355256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.7.194.253:35843)
0619 02:34:59.355753 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:59.355761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:59.355775 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:59.355824 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:59.355830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:59.355869 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:59.355946 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:59.356201 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.356949 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.358247 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.358729 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.358826 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.358871 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:59.359079 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 02:34:59.359137 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":96,"threads_started":1}
I20260619 02:34:59.359566 15384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } }
I20260619 02:34:59.359732 14683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91e26bb975234ecaadc52946641ecd68" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "905b3b91feb74d8ca2904dea1064bedd" is_pre_election: true
I20260619 02:34:59.359889 15384 leader_election.cc:288] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [CANDIDATE]: Term 2 election: Requested vote from peers 71ef3db455d5435da039949bfaa1ac86 (127.7.194.254:45193), 91e26bb975234ecaadc52946641ecd68 (127.7.194.252:37941)
I20260619 02:34:59.359898 14683 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91e26bb975234ecaadc52946641ecd68 in current term 2: Already voted for candidate 905b3b91feb74d8ca2904dea1064bedd in this term.
I20260619 02:34:59.360280 14849 leader_election.cc:302] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91e26bb975234ecaadc52946641ecd68; no voters: 71ef3db455d5435da039949bfaa1ac86, 905b3b91feb74d8ca2904dea1064bedd
I20260619 02:34:59.360359 14865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "905b3b91feb74d8ca2904dea1064bedd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91e26bb975234ecaadc52946641ecd68"
I20260619 02:34:59.360450 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.360038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:40394)
0619 02:34:59.360308 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.360373 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260619 02:34:59.360496 14865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:34:59.361038 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.360825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:40400)
0619 02:34:59.360936 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.360985 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:34:59.361135 14659 leader_election.cc:334] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71ef3db455d5435da039949bfaa1ac86 (127.7.194.254:45193): Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
I20260619 02:34:59.361618 14865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905b3b91feb74d8ca2904dea1064bedd in term 2.
I20260619 02:34:59.361732 15389 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:34:59.361955 14659 leader_election.cc:302] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 905b3b91feb74d8ca2904dea1064bedd, 91e26bb975234ecaadc52946641ecd68; no voters: 71ef3db455d5435da039949bfaa1ac86
I20260619 02:34:59.362109 15384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:34:59.362315 15384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [term 2 LEADER]: Becoming Leader. State: Replica: 905b3b91feb74d8ca2904dea1064bedd, State: Running, Role: LEADER
I20260619 02:34:59.362581 15384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } }
I20260619 02:34:59.363577 15394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905b3b91feb74d8ca2904dea1064bedd. Latest consensus state: current_term: 2 leader_uuid: "905b3b91feb74d8ca2904dea1064bedd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:59.363667 15394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:59.363973 15396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:34:59.366634 15396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:34:59.367064 15396 catalog_manager.cc:1259] Loaded cluster ID: e6c6f9ee024b4211a751114d0a536d6e
I20260619 02:34:59.367116 15396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:34:59.367690 15396 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:34:59.368095 15396 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Loaded TSK: 0
I20260619 02:34:59.368813 15396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:34:59.413777 14672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:34:59.416889 14672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:34:59.420176 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.419926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:40418)
0619 02:34:59.420046 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.420107 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:34:59.420204 14865 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [term 2 FOLLOWER]: Refusing update from remote peer 905b3b91feb74d8ca2904dea1064bedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 02:34:59.420903 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.420654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.0.0.1:40434)
0619 02:34:59.420758 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.420817 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 02:34:59.420915 15394 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 02:34:59.421100 14659 consensus_peers.cc:487] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd -> Peer 71ef3db455d5435da039949bfaa1ac86 (127.7.194.254:45193): Couldn't send request to peer 71ef3db455d5435da039949bfaa1ac86. Status: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:34:59.422533 15389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905b3b91feb74d8ca2904dea1064bedd. Latest consensus state: current_term: 2 leader_uuid: "905b3b91feb74d8ca2904dea1064bedd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:59.422669 15389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:34:59.423724 15394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "905b3b91feb74d8ca2904dea1064bedd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:59.423856 15394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd [sys.catalog]: This master's current role is: LEADER
I20260619 02:34:59.424506 15389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "905b3b91feb74d8ca2904dea1064bedd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "71ef3db455d5435da039949bfaa1ac86" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 45193 } } peers { permanent_uuid: "905b3b91feb74d8ca2904dea1064bedd" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "91e26bb975234ecaadc52946641ecd68" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 37941 } attrs { promote: false } } }
I20260619 02:34:59.424646 15389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e26bb975234ecaadc52946641ecd68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:34:59.434278 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.429694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:40641 (local address 127.0.0.1:57222)
0619 02:34:59.429859 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.429877 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:59.429962 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:59.430564 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:59.430568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:59.430584 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:59.430691 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.430698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.431855 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.431858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.432932 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.432940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.433840 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.433844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.433937 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.433970 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:34:59.434012 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:34:59.434131 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":45}
I20260619 02:34:59.434438 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.430216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:42707 (local address 127.0.0.1:37130)
0619 02:34:59.430453 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.430467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:59.430557 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:59.430819 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:59.430821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:59.430835 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:59.430906 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.430911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.432003 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.432009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.433080 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.433091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.433989 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.433993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.434102 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.434127 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:34:59.434175 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:34:59.434238 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260619 02:34:59.435027 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.430350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.7.194.193:42707)
0619 02:34:59.430591 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:59.430595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:59.430609 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:59.430657 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:59.430659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:59.430702 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:59.431004 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:59.431106 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.431882 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.433215 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.433852 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.434668 (+ 816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.434755 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:59.434802 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 02:34:59.434879 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20260619 02:34:59.435161 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.429966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:40645 (local address 127.0.0.1:40782)
0619 02:34:59.430094 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.430105 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:59.430162 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:59.431417 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:59.431423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:59.431442 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:59.431548 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.431553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.432641 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.432645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.433817 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.433827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.434757 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.434760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.434858 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.434900 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:34:59.434973 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:34:59.435037 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:34:59.435447 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.430550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.7.194.194:40645)
0619 02:34:59.430786 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:59.430790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:59.430804 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:34:59.431007 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:59.431010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:59.431266 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:59.431640 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:59.431736 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.432536 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.433980 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.434646 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.434745 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.434805 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:59.435000 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:34:59.435093 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":97,"threads_started":1}
I20260619 02:34:59.435557 14976 tablet_service.cc:1467] Processing CreateTablet for tablet ca22394cd7ea4380862ce7d93f9600ed (DEFAULT_TABLE table=test-table [id=eddb2fc91f714dd88aa0862050665948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:59.435704 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.429797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.7.194.195:40641)
0619 02:34:59.430309 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:59.430314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:59.430329 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:34:59.430379 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:59.430382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:59.430435 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:59.430796 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:59.430912 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.431750 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.433044 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.433699 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.434980 (+ 1281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.435042 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:59.435376 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 02:34:59.435463 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":97,"threads_started":1}
I20260619 02:34:59.435886 14976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca22394cd7ea4380862ce7d93f9600ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:59.435951 15113 tablet_service.cc:1467] Processing CreateTablet for tablet ca22394cd7ea4380862ce7d93f9600ed (DEFAULT_TABLE table=test-table [id=eddb2fc91f714dd88aa0862050665948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:59.436235 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca22394cd7ea4380862ce7d93f9600ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:59.436177 15249 tablet_service.cc:1467] Processing CreateTablet for tablet ca22394cd7ea4380862ce7d93f9600ed (DEFAULT_TABLE table=test-table [id=eddb2fc91f714dd88aa0862050665948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:34:59.436455 15249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca22394cd7ea4380862ce7d93f9600ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:34:59.438526 15407 tablet_bootstrap.cc:492] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472: Bootstrap starting.
I20260619 02:34:59.438930 15409 tablet_bootstrap.cc:492] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: Bootstrap starting.
I20260619 02:34:59.439131 15408 tablet_bootstrap.cc:492] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7: Bootstrap starting.
I20260619 02:34:59.439793 15407 tablet_bootstrap.cc:654] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:59.439888 15409 tablet_bootstrap.cc:654] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:59.440038 15408 tablet_bootstrap.cc:654] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7: Neither blocks nor log segments found. Creating new log.
I20260619 02:34:59.441114 15409 tablet_bootstrap.cc:492] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: No bootstrap required, opened a new log
I20260619 02:34:59.441159 15408 tablet_bootstrap.cc:492] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7: No bootstrap required, opened a new log
I20260619 02:34:59.441184 15409 ts_tablet_manager.cc:1397] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:34:59.441239 15408 ts_tablet_manager.cc:1397] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:34:59.441459 15407 tablet_bootstrap.cc:492] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472: No bootstrap required, opened a new log
I20260619 02:34:59.441531 15407 ts_tablet_manager.cc:1397] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:34:59.441491 15409 raft_consensus.cc:348] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } }
I20260619 02:34:59.441653 15409 raft_consensus.cc:374] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:59.441686 15409 raft_consensus.cc:729] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff80a9a53a84a5f9a8b60a334680d98, State: Initialized, Role: FOLLOWER
I20260619 02:34:59.441721 15408 raft_consensus.cc:348] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } }
I20260619 02:34:59.441830 15408 raft_consensus.cc:374] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:59.441869 15408 raft_consensus.cc:729] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5e2361000406e892e89dac775c4b7, State: Initialized, Role: FOLLOWER
I20260619 02:34:59.441852 15409 consensus_queue.cc:260] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } }
I20260619 02:34:59.441996 15408 consensus_queue.cc:260] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } }
I20260619 02:34:59.442015 15407 raft_consensus.cc:348] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } }
I20260619 02:34:59.442111 15407 raft_consensus.cc:374] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:34:59.442155 15407 raft_consensus.cc:729] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b090dd45ea8e4f3294be929815ed3472, State: Initialized, Role: FOLLOWER
I20260619 02:34:59.442230 15409 ts_tablet_manager.cc:1428] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:59.442348 15408 ts_tablet_manager.cc:1428] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:34:59.442322 15407 consensus_queue.cc:260] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } }
I20260619 02:34:59.442780 15407 ts_tablet_manager.cc:1428] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:34:59.443652 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.443256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.193:33515)
0619 02:34:59.443473 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.443541 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260619 02:34:59.444475 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.444086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.194:46555)
0619 02:34:59.444326 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.444381 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
I20260619 02:34:59.445155 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.444938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.194:44259)
0619 02:34:59.445027 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.445085 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 02:34:59.445825 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.445605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.193:50775)
0619 02:34:59.445695 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.445740 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 02:34:59.447064 15317 heartbeater.cc:503] Master 127.7.194.253:35843 was elected leader, sending a full tablet report...
I20260619 02:34:59.447254 15181 heartbeater.cc:503] Master 127.7.194.253:35843 was elected leader, sending a full tablet report...
I20260619 02:34:59.447685 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.442697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.195:59623)
0619 02:34:59.447539 (+ 4842us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.447600 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4780,"thread_start_us":90,"threads_started":1}
I20260619 02:34:59.447743 15044 heartbeater.cc:503] Master 127.7.194.253:35843 was elected leader, sending a full tablet report...
I20260619 02:34:59.449077 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.448835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.195:43457)
0619 02:34:59.448987 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.449034 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 02:34:59.467119 15182 tablet.cc:2369] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:34:59.595399 15415 raft_consensus.cc:482] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:34:59.595600 15415 raft_consensus.cc:504] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } }
I20260619 02:34:59.596916 15415 leader_election.cc:288] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ff80a9a53a84a5f9a8b60a334680d98 (127.7.194.195:40641), baa5e2361000406e892e89dac775c4b7 (127.7.194.193:42707)
I20260619 02:34:59.600874 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.597202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:40641 (local address 127.7.194.194:43813)
0619 02:34:59.597459 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.597475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:59.597624 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:59.597902 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:59.597905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:59.597924 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:59.598084 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.598091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.598851 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.598856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.599752 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.599762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.600428 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.600432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.600569 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.600586 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:34:59.600678 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:34:59.600740 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":74}
I20260619 02:34:59.600874 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.597365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:50283 (local address 127.7.194.193:42707)
0619 02:34:59.597482 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:59.597488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:59.597785 (+ 297us) server_negotiation.cc:407] Connection header received
0619 02:34:59.597902 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:59.597905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:59.597950 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:59.598030 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:59.598282 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.598767 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.599881 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.600275 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.600403 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.600459 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:59.600644 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 02:34:59.600729 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:34:59.600898 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.597312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:43813 (local address 127.7.194.195:40641)
0619 02:34:59.597454 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:59.597459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:59.597564 (+ 105us) server_negotiation.cc:407] Connection header received
0619 02:34:59.597739 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:59.597745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:59.597802 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:59.597879 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:34:59.598192 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.598734 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.599879 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.600281 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.600404 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.600460 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:59.600708 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 02:34:59.600785 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:34:59.600925 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.597218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:42707 (local address 127.7.194.194:50283)
0619 02:34:59.597677 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.597691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:34:59.597802 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:34:59.598055 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:34:59.598057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:34:59.598070 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:34:59.598182 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.598188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.598907 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.598910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.599754 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:34:59.599762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.600389 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.600392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:34:59.600490 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.600546 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 02:34:59.600612 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:34:59.600700 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":64,"thread_start_us":144,"threads_started":1}
I20260619 02:34:59.601426 15269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca22394cd7ea4380862ce7d93f9600ed" candidate_uuid: "b090dd45ea8e4f3294be929815ed3472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" is_pre_election: true
I20260619 02:34:59.601475 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca22394cd7ea4380862ce7d93f9600ed" candidate_uuid: "b090dd45ea8e4f3294be929815ed3472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e2361000406e892e89dac775c4b7" is_pre_election: true
I20260619 02:34:59.601641 14996 raft_consensus.cc:2454] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b090dd45ea8e4f3294be929815ed3472 in term 0.
I20260619 02:34:59.601696 15269 raft_consensus.cc:2454] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b090dd45ea8e4f3294be929815ed3472 in term 0.
I20260619 02:34:59.602097 15067 leader_election.cc:302] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ff80a9a53a84a5f9a8b60a334680d98, b090dd45ea8e4f3294be929815ed3472, baa5e2361000406e892e89dac775c4b7; no voters:
I20260619 02:34:59.602370 15415 raft_consensus.cc:2781] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:34:59.602423 15415 raft_consensus.cc:482] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:34:59.602452 15415 raft_consensus.cc:3037] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:59.603478 15415 raft_consensus.cc:504] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } }
I20260619 02:34:59.603826 15415 leader_election.cc:288] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [CANDIDATE]: Term 1 election: Requested vote from peers 4ff80a9a53a84a5f9a8b60a334680d98 (127.7.194.195:40641), baa5e2361000406e892e89dac775c4b7 (127.7.194.193:42707)
I20260619 02:34:59.604224 15269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca22394cd7ea4380862ce7d93f9600ed" candidate_uuid: "b090dd45ea8e4f3294be929815ed3472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff80a9a53a84a5f9a8b60a334680d98"
I20260619 02:34:59.604224 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca22394cd7ea4380862ce7d93f9600ed" candidate_uuid: "b090dd45ea8e4f3294be929815ed3472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e2361000406e892e89dac775c4b7"
I20260619 02:34:59.604362 14996 raft_consensus.cc:3037] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:59.604362 15269 raft_consensus.cc:3037] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:34:59.605530 14996 raft_consensus.cc:2454] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b090dd45ea8e4f3294be929815ed3472 in term 1.
I20260619 02:34:59.605589 15269 raft_consensus.cc:2454] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b090dd45ea8e4f3294be929815ed3472 in term 1.
I20260619 02:34:59.605847 15067 leader_election.cc:302] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b090dd45ea8e4f3294be929815ed3472, baa5e2361000406e892e89dac775c4b7; no voters:
I20260619 02:34:59.606047 15415 raft_consensus.cc:2781] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:34:59.606281 15415 raft_consensus.cc:686] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 1 LEADER]: Becoming Leader. State: Replica: b090dd45ea8e4f3294be929815ed3472, State: Running, Role: LEADER
I20260619 02:34:59.606549 15415 consensus_queue.cc:237] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } }
I20260619 02:34:59.606963 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.606723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.194:53573)
0619 02:34:59.606839 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.606898 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:34:59.608958 14671 catalog_manager.cc:5495] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 reported cstate change: term changed from 0 to 1, leader changed from <none> to b090dd45ea8e4f3294be929815ed3472 (127.7.194.194). New cstate: current_term: 1 leader_uuid: "b090dd45ea8e4f3294be929815ed3472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b090dd45ea8e4f3294be929815ed3472" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 } health_report { overall_health: UNKNOWN } } }
I20260619 02:34:59.630074 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.626521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.7.194.194:40645)
0619 02:34:59.626650 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:34:59.626655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:34:59.626671 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:34:59.626764 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:34:59.626766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:34:59.626825 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:34:59.626932 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:34:59.627167 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.627684 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.628531 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:34:59.628748 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:34:59.628865 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:34:59.629130 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:34:59.629674 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:34:59.629798 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:34:59.629856 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 02:34:59.629915 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:34:59.632988 14996 raft_consensus.cc:1261] T ca22394cd7ea4380862ce7d93f9600ed P baa5e2361000406e892e89dac775c4b7 [term 1 FOLLOWER]: Refusing update from remote peer b090dd45ea8e4f3294be929815ed3472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:59.633189 15269 raft_consensus.cc:1261] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [term 1 FOLLOWER]: Refusing update from remote peer b090dd45ea8e4f3294be929815ed3472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:34:59.633572 15415 consensus_queue.cc:1035] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5e2361000406e892e89dac775c4b7" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 42707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:59.633816 15420 consensus_queue.cc:1035] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff80a9a53a84a5f9a8b60a334680d98" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:34:59.635672 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.635402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.195:46881)
0619 02:34:59.635521 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.635580 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:34:59.635970 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.635681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.193:50013)
0619 02:34:59.635804 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.635885 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 02:34:59.638507 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.638016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.194:41111)
0619 02:34:59.638359 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.638421 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262}
I20260619 02:34:59.639729 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.639498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.193:54713)
0619 02:34:59.639602 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.639657 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 02:34:59.642891 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.642646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.195:40033)
0619 02:34:59.642766 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.642824 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:34:59.643496 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.642297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:45193 (local address 127.7.194.194:39661)
0619 02:34:59.643365 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.643419 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1000}
I20260619 02:34:59.649345 14672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:59.649521 14672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:34:59.653403 14672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 905b3b91feb74d8ca2904dea1064bedd: Sending DeleteTablet for 3 replicas of tablet ca22394cd7ea4380862ce7d93f9600ed
I20260619 02:34:59.654124 15249 tablet_service.cc:1514] Processing DeleteTablet for tablet ca22394cd7ea4380862ce7d93f9600ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:59 UTC) from {username='oryx', principal='oryx/127.7.194.253@KRBTEST.COM'} at 127.0.0.1:57222
I20260619 02:34:59.654587 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet ca22394cd7ea4380862ce7d93f9600ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:59 UTC) from {username='oryx', principal='oryx/127.7.194.253@KRBTEST.COM'} at 127.0.0.1:40782
I20260619 02:34:59.654640 15432 tablet_replica.cc:330] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: stopping tablet replica
I20260619 02:34:59.654786 14976 tablet_service.cc:1514] Processing DeleteTablet for tablet ca22394cd7ea4380862ce7d93f9600ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:34:59 UTC) from {username='oryx', principal='oryx/127.7.194.253@KRBTEST.COM'} at 127.0.0.1:37130
I20260619 02:34:59.655066 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 14817
I20260619 02:34:59.655192 15432 raft_consensus.cc:2229] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:34:59.655346 15432 raft_consensus.cc:2258] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:59.656338 15432 ts_tablet_manager.cc:1905] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:34:59.659032 15432 ts_tablet_manager.cc:1918] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:34:59.659123 15432 log.cc:1196] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/ca22394cd7ea4380862ce7d93f9600ed
I20260619 02:34:59.659431 15432 ts_tablet_manager.cc:1939] T ca22394cd7ea4380862ce7d93f9600ed P 4ff80a9a53a84a5f9a8b60a334680d98: Deleting consensus metadata
I20260619 02:34:59.660220 14660 catalog_manager.cc:4841] TS 4ff80a9a53a84a5f9a8b60a334680d98 (127.7.194.195:40641): tablet ca22394cd7ea4380862ce7d93f9600ed (table test-table [id=eddb2fc91f714dd88aa0862050665948]) successfully deleted
I20260619 02:34:59.664516 15433 tablet_replica.cc:330] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472: stopping tablet replica
I20260619 02:34:59.664709 15433 raft_consensus.cc:2229] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:34:59.665185 15433 raft_consensus.cc:2258] T ca22394cd7ea4380862ce7d93f9600ed P b090dd45ea8e4f3294be929815ed3472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:34:59.665239 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.664923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:42707 (local address 127.0.0.1:37138)
0619 02:34:59.665095 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.665146 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 02:34:59.665488 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 15050
W20260619 02:34:59.665534 14658 catalog_manager.cc:4584] TS baa5e2361000406e892e89dac775c4b7 (127.7.194.193:42707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca22394cd7ea4380862ce7d93f9600ed: Network error: Client connection negotiation failed: client connection to 127.7.194.193:42707: connect: Connection refused (error 111)
I20260619 02:34:59.671710 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 15186
I20260619 02:34:59.671917 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:34:59.671608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:40645 (local address 127.0.0.1:40800)
0619 02:34:59.671745 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:34:59.671813 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.194:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:34:59.678078 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 14536
I20260619 02:34:59.683931 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 14637
2026-06-19T02:34:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5429 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:34:59 dist-test-slave-9tkz krb5kdc[15437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:34:59 dist-test-slave-9tkz krb5kdc[15437](info): set up 2 sockets
Jun 19 02:34:59 dist-test-slave-9tkz krb5kdc[15437](info): commencing operation
krb5kdc: starting...
W20260619 02:35:00.726203 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:35:00 dist-test-slave-9tkz krb5kdc[15437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:35:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.254.keytab.
Entry for principal oryx/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.254.keytab.
I20260619 02:35:00.852569 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:40457
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.254.keytab
--principal=oryx/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:40457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:00.968461 15450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:00.968760 15450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:00.968803 15450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:00.973067 15450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:35:00.973143 15450 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:35:00.973161 15450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:00.973181 15450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:00.973196 15450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:35:00.973253 15450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:35:00.979138 15450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:40457
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.254.keytab
--principal=oryx/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:40457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:00.980450 15450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:00.981653 15450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:00.990499 15456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:00.990510 15455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:00.990500 15458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:00.990757 15450 server_base.cc:1034] running on GCE node
I20260619 02:35:00.991508 15450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:00.992785 15450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:00.994004 15450 hybrid_clock.cc:648] HybridClock initialized: now 1781836500993961 us; error 51 us; skew 500 ppm
Jun 19 02:35:00 dist-test-slave-9tkz krb5kdc[15437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836500, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:00.997896 15450 init.cc:377] Logged in from keytab as oryx/127.7.194.254@KRBTEST.COM (short username oryx)
I20260619 02:35:00.999019 15450 webserver.cc:460] Webserver started at http://127.7.194.254:37711/ using document root <none> and password file <none>
I20260619 02:35:00.999686 15450 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:00.999773 15450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:01.000106 15450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:01.002413 15450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "5a37c65704194797a7c19a36de2b6fe1"
format_stamp: "Formatted at 2026-06-19 02:35:00 on dist-test-slave-9tkz"
server_key: "bcebe2a8b45e136fcd7b53900f413b91"
server_key_iv: "c3f398ca91c896defe4ea16e2127c3cd"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.003019 15450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "5a37c65704194797a7c19a36de2b6fe1"
format_stamp: "Formatted at 2026-06-19 02:35:00 on dist-test-slave-9tkz"
server_key: "bcebe2a8b45e136fcd7b53900f413b91"
server_key_iv: "c3f398ca91c896defe4ea16e2127c3cd"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.007027 15450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:35:01.009687 15465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:01.011108 15450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:35:01.011271 15450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "5a37c65704194797a7c19a36de2b6fe1"
format_stamp: "Formatted at 2026-06-19 02:35:00 on dist-test-slave-9tkz"
server_key: "bcebe2a8b45e136fcd7b53900f413b91"
server_key_iv: "c3f398ca91c896defe4ea16e2127c3cd"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.011401 15450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:01.037173 15450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:01.040879 15450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:01.041097 15450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:01.049150 15450 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:40457
I20260619 02:35:01.049199 15527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:40457 every 8 connection(s)
I20260619 02:35:01.050220 15450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:35:01.053977 15528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:01.059484 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 15450
I20260619 02:35:01.059625 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:35:01.059903 7947 external_mini_cluster.cc:1428] Setting key 96c1c8829e743945e75179ba256b11bb
I20260619 02:35:01.061909 15528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: Bootstrap starting.
I20260619 02:35:01.064548 15528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:01.065326 15528 log.cc:824] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:01.067644 15528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: No bootstrap required, opened a new log
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:35:01.071586 15528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37c65704194797a7c19a36de2b6fe1" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40457 } }
I20260619 02:35:01.071863 15528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:01.071923 15528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a37c65704194797a7c19a36de2b6fe1, State: Initialized, Role: FOLLOWER
I20260619 02:35:01.072383 15528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37c65704194797a7c19a36de2b6fe1" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40457 } }
I20260619 02:35:01.072540 15528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:35:01.072623 15528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:35:01.072738 15528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:01.074014 15528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37c65704194797a7c19a36de2b6fe1" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40457 } }
I20260619 02:35:01.074429 15528 leader_election.cc:302] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a37c65704194797a7c19a36de2b6fe1; no voters:
I20260619 02:35:01.074793 15528 leader_election.cc:288] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:35:01.074939 15533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:01.075215 15533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a37c65704194797a7c19a36de2b6fe1, State: Running, Role: LEADER
I20260619 02:35:01.075544 15533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37c65704194797a7c19a36de2b6fe1" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40457 } }
I20260619 02:35:01.075786 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.061460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.7.194.254:40457)
0619 02:35:01.061925 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.061941 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.061979 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:35:01.062668 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.062687 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.063035 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.063331 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:01.064243 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.065387 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.066503 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.067004 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.067138 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.070287 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:01.070293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:01.070308 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:01.070337 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:01.072277 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.072792 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.072799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.072807 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.072963 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.073222 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.073226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.073227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.073830 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:01.074053 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.074414 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.074748 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":125,"threads_started":1}
I20260619 02:35:01.076122 15528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:35:01.078297 15534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a37c65704194797a7c19a36de2b6fe1. Latest consensus state: current_term: 1 leader_uuid: "5a37c65704194797a7c19a36de2b6fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37c65704194797a7c19a36de2b6fe1" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40457 } } }
I20260619 02:35:01.078779 15534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:01.078738 15535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a37c65704194797a7c19a36de2b6fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37c65704194797a7c19a36de2b6fe1" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 40457 } } }
I20260619 02:35:01.079048 15535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1 [sys.catalog]: This master's current role is: LEADER
W20260619 02:35:01.081425 15547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:35:01.081496 15547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:35:01.081589 15549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:35:01.082348 15549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:35:01.088011 15549 catalog_manager.cc:1347] Generated new cluster ID: 9b4cd0e39f6f4ac6add28bdf3a232ac8
I20260619 02:35:01.088090 15549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:35:01.098925 15549 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:35:01.099990 15549 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:35:01.106751 15549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: Generated new TSK 0
I20260619 02:35:01.107663 15549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab.
Entry for principal oryx/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab.
I20260619 02:35:01.144744 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:40457
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab
--principal=oryx/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:01.261317 15554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:01.261691 15554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:01.261783 15554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:01.265921 15554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:01.266033 15554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:01.266112 15554 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:35:01.266193 15554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:35:01.271854 15554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab
--principal=oryx/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.194.254:40457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:01.273072 15554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:01.274222 15554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:01.284425 15559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:01.284471 15562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:01.284478 15560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:01.284811 15554 server_base.cc:1034] running on GCE node
I20260619 02:35:01.285257 15554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:01.285981 15554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:01.287156 15554 hybrid_clock.cc:648] HybridClock initialized: now 1781836501287125 us; error 54 us; skew 500 ppm
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836501, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:01.290055 15554 init.cc:377] Logged in from keytab as oryx/127.7.194.193@KRBTEST.COM (short username oryx)
I20260619 02:35:01.291041 15554 webserver.cc:460] Webserver started at http://127.7.194.193:39467/ using document root <none> and password file <none>
I20260619 02:35:01.291628 15554 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:01.291715 15554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:01.291946 15554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:01.294160 15554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "c19da35034964e16bb46cb6ecbc8b948"
format_stamp: "Formatted at 2026-06-19 02:35:01 on dist-test-slave-9tkz"
server_key: "bf8aa8d864baad374465067c3e48b83f"
server_key_iv: "f18201ae2d0c2c06f03705b42b526ddc"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.294739 15554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "c19da35034964e16bb46cb6ecbc8b948"
format_stamp: "Formatted at 2026-06-19 02:35:01 on dist-test-slave-9tkz"
server_key: "bf8aa8d864baad374465067c3e48b83f"
server_key_iv: "f18201ae2d0c2c06f03705b42b526ddc"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.298651 15554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:35:01.301257 15569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:01.302627 15554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:01.302767 15554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "c19da35034964e16bb46cb6ecbc8b948"
format_stamp: "Formatted at 2026-06-19 02:35:01 on dist-test-slave-9tkz"
server_key: "bf8aa8d864baad374465067c3e48b83f"
server_key_iv: "f18201ae2d0c2c06f03705b42b526ddc"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.302888 15554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:01.323818 15554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:01.327011 15554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:01.327234 15554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:01.328819 15554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:01.328888 15554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:01.328960 15554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:01.329011 15554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:01.339811 15554 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:41111
I20260619 02:35:01.339838 15684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:41111 every 8 connection(s)
I20260619 02:35:01.340749 15554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:35:01.341665 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 15554
I20260619 02:35:01.341806 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:35:01.342175 7947 external_mini_cluster.cc:1428] Setting key 95a082f24e90871d6e4f2c5614629215
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836501, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:35:01.347656 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.329845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:34243 (local address 127.7.194.254:40457)
0619 02:35:01.330028 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.330036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.331037 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 02:35:01.332446 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.332453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.332524 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.332623 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:01.333966 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.334558 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.336450 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.336775 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.336907 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.343537 (+ 6630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:01.343541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:01.343544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:01.343569 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:01.345137 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.345733 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.345739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.345742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.345847 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.346258 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.346261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.346263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.346536 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:01.346643 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.347238 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.347393 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:35:01.348419 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.330324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40457 (local address 127.7.194.193:34243)
0619 02:35:01.330836 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:01.330901 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:01.332165 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:01.332804 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:01.332812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:01.333162 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:01.333815 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.333833 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.334686 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.334693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.336314 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.336328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.336961 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.336965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.337156 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.341415 (+ 4259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:01.341440 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:01.343373 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:01.345272 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.345278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.345292 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.345620 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.345964 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.345967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.345968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.346146 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.346651 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:01.346658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:01.346936 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 02:35:01.347138 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 02:35:01.347434 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260619 02:35:01.349886 15685 heartbeater.cc:347] Connected to a master server at 127.7.194.254:40457
I20260619 02:35:01.350075 15685 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:01.350617 15685 heartbeater.cc:511] Master 127.7.194.254:40457 requested a full tablet report, sending...
I20260619 02:35:01.352197 15481 ts_manager.cc:194] Registered new tserver with Master: c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111)
WARNING: no policy specified for oryx/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:35:01.353789 15481 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:34243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.194@KRBTEST.COM" created.
I20260619 02:35:01.361104 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.352533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40457 (local address 127.7.194.193:42951)
0619 02:35:01.352937 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:01.352949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:01.353035 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:01.353268 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:01.353270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:01.353430 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:01.353627 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.353635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.354599 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.354601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.355449 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.355462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.356034 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.356041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.356177 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.356730 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:01.356750 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:01.357176 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:01.359306 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.359309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.359311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.359526 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.359971 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.359974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.359975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.360091 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.360578 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:01.360580 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:01.360679 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 02:35:01.360772 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:35:01.360894 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":15,"thread_start_us":121,"threads_started":1}
I20260619 02:35:01.361280 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.352635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:42951 (local address 127.7.194.254:40457)
0619 02:35:01.352765 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.352772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.353032 (+ 260us) server_negotiation.cc:407] Connection header received
0619 02:35:01.353122 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.353125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.353172 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.353262 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:01.353744 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.354464 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.355588 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.355894 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.356009 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.357318 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:01.357322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:01.357325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:01.357352 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:01.359180 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.359751 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.359758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.359763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.359862 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.360185 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.360188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.360190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.360476 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:01.360596 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.360864 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.361025 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab.
Entry for principal oryx/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab.
I20260619 02:35:01.373416 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:40457
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab
--principal=oryx/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:01.489461 15694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:01.489823 15694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:01.489872 15694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:01.494002 15694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:01.494066 15694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:01.494108 15694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:35:01.494143 15694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:35:01.499802 15694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab
--principal=oryx/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.194.254:40457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:01.500892 15694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:01.502024 15694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:01.512254 15702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:01.512264 15699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:01.512306 15700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:01.512465 15694 server_base.cc:1034] running on GCE node
I20260619 02:35:01.513031 15694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:01.513852 15694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:01.515053 15694 hybrid_clock.cc:648] HybridClock initialized: now 1781836501515033 us; error 32 us; skew 500 ppm
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836501, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:01.518184 15694 init.cc:377] Logged in from keytab as oryx/127.7.194.194@KRBTEST.COM (short username oryx)
I20260619 02:35:01.519186 15694 webserver.cc:460] Webserver started at http://127.7.194.194:45415/ using document root <none> and password file <none>
I20260619 02:35:01.519797 15694 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:01.519887 15694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:01.520126 15694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:01.522344 15694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "e8c600c8d984495c94dedd0d10707961"
format_stamp: "Formatted at 2026-06-19 02:35:01 on dist-test-slave-9tkz"
server_key: "d2b8d038487dd376104c9f491a8bce4e"
server_key_iv: "240059960aa6a9e8e8c6bcb358780024"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.522943 15694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "e8c600c8d984495c94dedd0d10707961"
format_stamp: "Formatted at 2026-06-19 02:35:01 on dist-test-slave-9tkz"
server_key: "d2b8d038487dd376104c9f491a8bce4e"
server_key_iv: "240059960aa6a9e8e8c6bcb358780024"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.528007 15694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:35:01.530787 15709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:01.532271 15694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:01.532428 15694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "e8c600c8d984495c94dedd0d10707961"
format_stamp: "Formatted at 2026-06-19 02:35:01 on dist-test-slave-9tkz"
server_key: "d2b8d038487dd376104c9f491a8bce4e"
server_key_iv: "240059960aa6a9e8e8c6bcb358780024"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.532569 15694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:01.563015 15694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:01.566362 15694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:01.566586 15694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:01.568184 15694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:01.568259 15694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:01.568326 15694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:01.568375 15694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836501, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:35:01.579843 15694 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:42437
I20260619 02:35:01.579972 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:42437 every 8 connection(s)
I20260619 02:35:01.580945 15694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:35:01.582679 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.568960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:46895 (local address 127.7.194.254:40457)
0619 02:35:01.569122 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.569128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.570191 (+ 1063us) server_negotiation.cc:407] Connection header received
0619 02:35:01.571601 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.571607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.571674 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.571794 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:01.573098 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.573725 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.575219 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.575456 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.575566 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.578492 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:01.578496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:01.578499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:01.578529 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:01.580286 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.580859 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.580862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.580863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.580981 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.581420 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.581425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.581427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.581730 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:01.581837 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.582430 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.582557 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:35:01.583508 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.569428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40457 (local address 127.7.194.194:46895)
0619 02:35:01.570006 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:01.570070 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:01.571281 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:01.571991 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:01.571998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:01.572354 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:01.572944 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.572968 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.573845 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.573848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.575097 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.575106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.575629 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.575633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.575855 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.576564 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:01.576587 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:01.578279 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:01.580395 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.580400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.580415 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.580756 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.581116 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.581122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.581125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.581334 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.581889 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:01.581895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:01.582187 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 02:35:01.582350 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 02:35:01.582627 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":170,"threads_started":1}
I20260619 02:35:01.584631 15825 heartbeater.cc:347] Connected to a master server at 127.7.194.254:40457
I20260619 02:35:01.584789 15825 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:01.585278 15825 heartbeater.cc:511] Master 127.7.194.254:40457 requested a full tablet report, sending...
I20260619 02:35:01.586342 15481 ts_manager.cc:194] Registered new tserver with Master: e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437)
I20260619 02:35:01.587061 15481 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:46895
I20260619 02:35:01.590830 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 15694
I20260619 02:35:01.590958 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:35:01.591305 7947 external_mini_cluster.cc:1428] Setting key f892fa126257f95c3a66b56330a1e464
I20260619 02:35:01.594507 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.586814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40457 (local address 127.7.194.194:60671)
0619 02:35:01.587103 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:01.587118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:01.587435 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:01.587687 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:01.587690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:01.587888 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:01.588020 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.588026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.588761 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.588764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.589336 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.589345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.589853 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.589857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.589962 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.590473 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:01.590485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:01.590977 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:01.592951 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.592954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.592957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.593179 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.593469 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.593472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.593473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.593594 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.594077 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:01.594079 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:01.594155 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 02:35:01.594234 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:35:01.594342 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":3,"thread_start_us":117,"threads_started":1}
I20260619 02:35:01.594688 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.586870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:60671 (local address 127.7.194.254:40457)
0619 02:35:01.586994 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.586998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.587192 (+ 194us) server_negotiation.cc:407] Connection header received
0619 02:35:01.587538 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.587541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.587584 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.587671 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:01.588129 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.588640 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.589454 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.589737 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.589843 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.591135 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:01.591140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:01.591145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:01.591171 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:01.592859 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.593271 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.593275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.593276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.593366 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.593671 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.593677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.593681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.593987 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:01.594094 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.594313 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.594442 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for oryx/127.7.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab.
Entry for principal oryx/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab.
I20260619 02:35:01.620085 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:40457
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab
--principal=oryx/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:01.736930 15834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:01.737272 15834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:01.737360 15834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:01.741570 15834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:01.741689 15834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:01.741773 15834 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:35:01.741871 15834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:35:01.747458 15834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab
--principal=oryx/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.194.254:40457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:01.748703 15834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:01.750000 15834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:01.760124 15839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:01.760124 15840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:01.760215 15842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:01.760356 15834 server_base.cc:1034] running on GCE node
I20260619 02:35:01.760881 15834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:01.761621 15834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:01.762856 15834 hybrid_clock.cc:648] HybridClock initialized: now 1781836501762779 us; error 52 us; skew 500 ppm
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836501, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:01.765784 15834 init.cc:377] Logged in from keytab as oryx/127.7.194.195@KRBTEST.COM (short username oryx)
I20260619 02:35:01.766752 15834 webserver.cc:460] Webserver started at http://127.7.194.195:36373/ using document root <none> and password file <none>
I20260619 02:35:01.767302 15834 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:01.767360 15834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:01.767530 15834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:01.769944 15834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "8dd4b6062261428ab17c019f64d3dc09"
format_stamp: "Formatted at 2026-06-19 02:35:01 on dist-test-slave-9tkz"
server_key: "f8270a49855d703df34cfa5d5b3fc171"
server_key_iv: "b8dab08177407a2333183e47d1a5b380"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.770506 15834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "8dd4b6062261428ab17c019f64d3dc09"
format_stamp: "Formatted at 2026-06-19 02:35:01 on dist-test-slave-9tkz"
server_key: "f8270a49855d703df34cfa5d5b3fc171"
server_key_iv: "b8dab08177407a2333183e47d1a5b380"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.774370 15834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:35:01.776980 15849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:01.778440 15834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:35:01.778558 15834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "8dd4b6062261428ab17c019f64d3dc09"
format_stamp: "Formatted at 2026-06-19 02:35:01 on dist-test-slave-9tkz"
server_key: "f8270a49855d703df34cfa5d5b3fc171"
server_key_iv: "b8dab08177407a2333183e47d1a5b380"
server_key_version: "encryptionkey@0"
I20260619 02:35:01.778668 15834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:01.804481 15834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:01.807909 15834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:01.808154 15834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:01.809804 15834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:01.809875 15834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:01.809947 15834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:01.809988 15834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836501, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:35:01.820883 15834 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:37487
I20260619 02:35:01.820966 15964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:37487 every 8 connection(s)
I20260619 02:35:01.821966 15834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:35:01.824401 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.810732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:56711 (local address 127.7.194.254:40457)
0619 02:35:01.810924 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.810929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.811819 (+ 890us) server_negotiation.cc:407] Connection header received
0619 02:35:01.813060 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.813063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.813124 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.813225 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:01.814603 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.815556 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.817252 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.817501 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.817645 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.820736 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:01.820739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:01.820741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:01.820768 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:01.822532 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.823053 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.823056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.823057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.823142 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.823409 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.823412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.823413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.823586 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:01.823662 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.824181 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.824280 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:35:01.825315 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.811195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40457 (local address 127.7.194.195:56711)
0619 02:35:01.811651 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:01.811703 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:01.812815 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:01.813418 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:01.813427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:01.813866 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:01.814429 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.814444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.815699 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.815703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.817123 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.817133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.817765 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.817769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.817946 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.818575 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:01.818602 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:01.820510 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:01.822670 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.822676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.822691 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.822969 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.823221 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.823223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.823225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.823340 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.823696 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:01.823702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:01.823957 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 02:35:01.824136 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 02:35:01.824412 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":127,"threads_started":1}
I20260619 02:35:01.826553 15965 heartbeater.cc:347] Connected to a master server at 127.7.194.254:40457
I20260619 02:35:01.826717 15965 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:01.827188 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 15834
I20260619 02:35:01.827288 15965 heartbeater.cc:511] Master 127.7.194.254:40457 requested a full tablet report, sending...
I20260619 02:35:01.827333 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:35:01.827718 7947 external_mini_cluster.cc:1428] Setting key d20d2063af775a17d966d0777115eb5b
I20260619 02:35:01.828495 15482 ts_manager.cc:194] Registered new tserver with Master: 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487)
I20260619 02:35:01.829319 15482 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:56711
I20260619 02:35:01.830082 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:35:01.836597 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.829167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40457 (local address 127.7.194.195:49519)
0619 02:35:01.829459 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:01.829475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:01.829608 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:01.830146 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:01.830148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:01.830335 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:01.830441 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.830446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.831197 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.831200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.831759 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.831767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.832297 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.832300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.832393 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.832942 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:01.832954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:01.833273 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:01.835004 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.835009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.835011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.835213 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.835497 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:01.835502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:01.835504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:01.835591 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:01.836028 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:01.836031 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:01.836135 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 02:35:01.836237 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:35:01.836396 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":110,"threads_started":1}
I20260619 02:35:01.836597 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.829724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:49519 (local address 127.7.194.254:40457)
0619 02:35:01.829847 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.829853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.829878 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:35:01.829914 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.829917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.829962 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.830035 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:01.830555 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.831073 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.831882 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.832189 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.832296 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.833416 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:01.833419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:01.833422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:01.833446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:01.834897 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.835310 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.835314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.835316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.835410 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.835683 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.835686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.835688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.835930 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:01.836040 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.836350 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.836470 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836501, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:35:01.855796 15531 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:01.849343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.7.194.254:40457)
0619 02:35:01.849482 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.849486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.849498 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:35:01.849539 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.849587 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.849635 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.849722 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:01.850563 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.851151 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.852008 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.852227 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.852330 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.855652 (+ 3322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58598: BlockingRecv error: recv got EOF from 127.0.0.1:58598 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 02:35:01 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836501, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:35:01.865224 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.857187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.7.194.254:40457)
0619 02:35:01.857298 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.857304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.857473 (+ 169us) server_negotiation.cc:407] Connection header received
0619 02:35:01.857673 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.857676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.857718 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.857797 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:01.858378 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.858885 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.859684 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.859899 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.860015 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.862251 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:01.862254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:01.862256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:01.862281 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:01.863588 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.864023 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.864027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.864029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.864108 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:01.864407 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:01.864410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:01.864411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:01.864587 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:01.864723 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.864946 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.865056 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:35:01.868270 15481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:35:01.871483 15481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:35:01.887050 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.880660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:37487 (local address 127.0.0.1:35618)
0619 02:35:01.881082 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:01.881137 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:01.881271 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:01.882528 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:01.882535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:01.882556 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:01.882713 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.882720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.884047 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.884051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.885402 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.885410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.886487 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.886491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.886592 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.886632 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 02:35:01.886750 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 02:35:01.886871 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260619 02:35:01.887089 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.880830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.7.194.195:37487)
0619 02:35:01.881850 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.881863 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.881892 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:35:01.881957 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.881961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.882333 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.882552 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:01.882879 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.883907 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.885516 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.886360 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.886480 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.886661 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.886792 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.886894 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":111,"threads_started":1}
I20260619 02:35:01.887219 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.880766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41111 (local address 127.0.0.1:37000)
0619 02:35:01.882089 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:01.882103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:01.882321 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:01.882977 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:01.882980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:01.883000 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:01.883107 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.883113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.884500 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.884503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.885655 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.885664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.886821 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.886825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.886911 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.886933 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:35:01.887004 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:35:01.887069 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"mutex_wait_us":19,"thread_start_us":87,"threads_started":1}
I20260619 02:35:01.887601 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.882141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:42437 (local address 127.0.0.1:38440)
0619 02:35:01.882451 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:01.882827 (+ 376us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:01.882908 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:01.883534 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:01.883537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:01.883553 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:01.883630 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.883635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.884992 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.884996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.886027 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:01.886034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.887180 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.887184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:01.887264 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.887285 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:35:01.887359 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:35:01.887462 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":24,"thread_start_us":105,"threads_started":1}
I20260619 02:35:01.887606 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.881346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.7.194.193:41111)
0619 02:35:01.881634 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.881641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.882248 (+ 607us) server_negotiation.cc:407] Connection header received
0619 02:35:01.882502 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.882508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.882705 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.882856 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:01.883206 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.884356 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.885781 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.886515 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.886623 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.886760 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.887381 (+ 621us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.887491 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260619 02:35:01.887996 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:01.882319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.7.194.194:42437)
0619 02:35:01.882864 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:01.882877 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:01.882899 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:35:01.883061 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:01.883065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:01.883254 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:01.883401 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:01.883749 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.884875 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.886138 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:01.886872 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:01.886973 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:01.887120 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:01.887753 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 02:35:01.887851 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":107,"threads_started":1}
I20260619 02:35:01.890637 15759 tablet_service.cc:1467] Processing CreateTablet for tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c (DEFAULT_TABLE table=test-table [id=41350ff8123d4e53aa860e8b26d7e3d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:01.890645 15899 tablet_service.cc:1467] Processing CreateTablet for tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c (DEFAULT_TABLE table=test-table [id=41350ff8123d4e53aa860e8b26d7e3d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:01.890643 15619 tablet_service.cc:1467] Processing CreateTablet for tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c (DEFAULT_TABLE table=test-table [id=41350ff8123d4e53aa860e8b26d7e3d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:01.891597 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:01.891619 15619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:01.891619 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:01.898267 15993 tablet_bootstrap.cc:492] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: Bootstrap starting.
I20260619 02:35:01.899580 15995 tablet_bootstrap.cc:492] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: Bootstrap starting.
I20260619 02:35:01.900727 15993 tablet_bootstrap.cc:654] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:01.901331 15995 tablet_bootstrap.cc:654] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:01.901513 15993 log.cc:824] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:01.901883 15994 tablet_bootstrap.cc:492] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: Bootstrap starting.
I20260619 02:35:01.902185 15995 log.cc:824] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:01.903587 15993 tablet_bootstrap.cc:492] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: No bootstrap required, opened a new log
I20260619 02:35:01.903795 15993 ts_tablet_manager.cc:1397] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:35:01.904099 15994 tablet_bootstrap.cc:654] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:01.904116 15995 tablet_bootstrap.cc:492] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: No bootstrap required, opened a new log
I20260619 02:35:01.904305 15995 ts_tablet_manager.cc:1397] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 02:35:01.904911 15994 log.cc:824] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:01.906878 15994 tablet_bootstrap.cc:492] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: No bootstrap required, opened a new log
I20260619 02:35:01.907083 15994 ts_tablet_manager.cc:1397] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:35:01.907325 15993 raft_consensus.cc:348] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } }
I20260619 02:35:01.907580 15993 raft_consensus.cc:374] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:01.907635 15993 raft_consensus.cc:729] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c600c8d984495c94dedd0d10707961, State: Initialized, Role: FOLLOWER
I20260619 02:35:01.908133 15993 consensus_queue.cc:260] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } }
I20260619 02:35:01.908111 15995 raft_consensus.cc:348] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } }
I20260619 02:35:01.908378 15995 raft_consensus.cc:374] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:01.908429 15995 raft_consensus.cc:729] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd4b6062261428ab17c019f64d3dc09, State: Initialized, Role: FOLLOWER
I20260619 02:35:01.908938 15995 consensus_queue.cc:260] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } }
I20260619 02:35:01.909204 15993 ts_tablet_manager.cc:1428] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:35:01.909485 15825 heartbeater.cc:503] Master 127.7.194.254:40457 was elected leader, sending a full tablet report...
I20260619 02:35:01.910080 15995 ts_tablet_manager.cc:1428] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260619 02:35:01.910416 15965 heartbeater.cc:503] Master 127.7.194.254:40457 was elected leader, sending a full tablet report...
I20260619 02:35:01.911780 15994 raft_consensus.cc:348] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } }
I20260619 02:35:01.912010 15994 raft_consensus.cc:374] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:01.912079 15994 raft_consensus.cc:729] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19da35034964e16bb46cb6ecbc8b948, State: Initialized, Role: FOLLOWER
I20260619 02:35:01.912492 15994 consensus_queue.cc:260] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } }
I20260619 02:35:01.913384 15994 ts_tablet_manager.cc:1428] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:35:01.913394 15685 heartbeater.cc:503] Master 127.7.194.254:40457 was elected leader, sending a full tablet report...
W20260619 02:35:02.073062 15966 tablet.cc:2369] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:35:02.082412 15826 tablet.cc:2369] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:35:02.092151 15686 tablet.cc:2369] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:35:02.179982 15999 raft_consensus.cc:482] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:35:02.180218 15999 raft_consensus.cc:504] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } }
I20260619 02:35:02.181429 15999 leader_election.cc:288] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487), c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111)
I20260619 02:35:02.186986 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.181844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:42783 (local address 127.7.194.193:41111)
0619 02:35:02.181986 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.181991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.182279 (+ 288us) server_negotiation.cc:407] Connection header received
0619 02:35:02.182348 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.182352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.182406 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.182494 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:02.182789 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.183617 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.185462 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.186158 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.186299 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.186359 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.186755 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.186846 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:35:02.186986 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.182111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:37039 (local address 127.7.194.195:37487)
0619 02:35:02.182285 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.182307 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.182324 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:35:02.182382 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.182387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.182434 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.182525 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:02.182874 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.183648 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.185462 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.186158 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.186299 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.186359 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.186755 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.186846 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:35:02.187017 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.181731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41111 (local address 127.7.194.194:42783)
0619 02:35:02.182066 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.182077 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.182140 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.182519 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.182522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.182538 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:02.182690 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.182695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.183770 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.183813 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.185270 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.185282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.186308 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.186312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.186548 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.186618 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.186703 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.186813 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":181,"spinlock_wait_cycles":2432,"thread_start_us":100,"threads_started":1}
I20260619 02:35:02.187017 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.181651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:37487 (local address 127.7.194.194:37039)
0619 02:35:02.181963 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.181982 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.182095 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.182534 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.182537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.182553 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:02.182713 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.182719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.183770 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.183774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.185270 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.185294 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.186308 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.186312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.186553 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.186618 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.186704 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.186813 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":243,"thread_start_us":112,"threads_started":1}
I20260619 02:35:02.187888 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d2d8a95d7b43d2a8f1d2e5b3663f2c" candidate_uuid: "e8c600c8d984495c94dedd0d10707961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd4b6062261428ab17c019f64d3dc09" is_pre_election: true
I20260619 02:35:02.187888 15639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d2d8a95d7b43d2a8f1d2e5b3663f2c" candidate_uuid: "e8c600c8d984495c94dedd0d10707961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19da35034964e16bb46cb6ecbc8b948" is_pre_election: true
I20260619 02:35:02.188244 15639 raft_consensus.cc:2454] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8c600c8d984495c94dedd0d10707961 in term 0.
I20260619 02:35:02.188244 15919 raft_consensus.cc:2454] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8c600c8d984495c94dedd0d10707961 in term 0.
I20260619 02:35:02.188737 15711 leader_election.cc:302] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dd4b6062261428ab17c019f64d3dc09, e8c600c8d984495c94dedd0d10707961; no voters:
I20260619 02:35:02.189023 15999 raft_consensus.cc:2781] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:35:02.189114 15999 raft_consensus.cc:482] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:35:02.189157 15999 raft_consensus.cc:3037] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.190323 15999 raft_consensus.cc:504] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } }
I20260619 02:35:02.190694 15999 leader_election.cc:288] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [CANDIDATE]: Term 1 election: Requested vote from peers 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487), c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111)
I20260619 02:35:02.191170 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d2d8a95d7b43d2a8f1d2e5b3663f2c" candidate_uuid: "e8c600c8d984495c94dedd0d10707961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd4b6062261428ab17c019f64d3dc09"
I20260619 02:35:02.191212 15639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d2d8a95d7b43d2a8f1d2e5b3663f2c" candidate_uuid: "e8c600c8d984495c94dedd0d10707961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19da35034964e16bb46cb6ecbc8b948"
I20260619 02:35:02.191322 15919 raft_consensus.cc:3037] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.191335 15639 raft_consensus.cc:3037] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.192397 15639 raft_consensus.cc:2454] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8c600c8d984495c94dedd0d10707961 in term 1.
I20260619 02:35:02.192716 15710 leader_election.cc:302] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19da35034964e16bb46cb6ecbc8b948, e8c600c8d984495c94dedd0d10707961; no voters:
I20260619 02:35:02.192752 15919 raft_consensus.cc:2454] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8c600c8d984495c94dedd0d10707961 in term 1.
I20260619 02:35:02.192909 15999 raft_consensus.cc:2781] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:02.193120 15999 raft_consensus.cc:686] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 1 LEADER]: Becoming Leader. State: Replica: e8c600c8d984495c94dedd0d10707961, State: Running, Role: LEADER
I20260619 02:35:02.193506 15999 consensus_queue.cc:237] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } }
I20260619 02:35:02.196705 15481 catalog_manager.cc:5495] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8c600c8d984495c94dedd0d10707961 (127.7.194.194). New cstate: current_term: 1 leader_uuid: "e8c600c8d984495c94dedd0d10707961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } health_report { overall_health: HEALTHY } } }
I20260619 02:35:02.257824 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.253951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.7.194.194:42437)
0619 02:35:02.254109 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.254114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.254129 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:35:02.254283 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.254285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.254342 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.254452 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:35:02.254718 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.255241 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.256203 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.256495 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.256585 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.256843 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:35:02.257310 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:35:02.257414 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.257593 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.257653 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:35:02.264034 15639 raft_consensus.cc:1261] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Refusing update from remote peer e8c600c8d984495c94dedd0d10707961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:02.264034 15919 raft_consensus.cc:1261] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Refusing update from remote peer e8c600c8d984495c94dedd0d10707961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:02.264730 16004 consensus_queue.cc:1035] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:02.264909 15999 consensus_queue.cc:1035] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:02.273706 16008 mvcc.cc:205] Tried to move back new op lower bound from 7298402313266249728 to 7298402312987545600. Current Snapshot: MvccSnapshot[applied={T|T < 7298402313266249728}]
I20260619 02:35:02.276186 16009 mvcc.cc:205] Tried to move back new op lower bound from 7298402313266249728 to 7298402312987545600. Current Snapshot: MvccSnapshot[applied={T|T < 7298402313266249728}]
I20260619 02:35:02.281260 15481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:02.281462 15481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:02.284236 15481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: Sending DeleteTablet for 3 replicas of tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c
I20260619 02:35:02.285090 15899 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:02 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:35618
I20260619 02:35:02.285115 15619 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:02 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:37000
I20260619 02:35:02.285156 15759 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:02 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:38440
I20260619 02:35:02.285739 16017 tablet_replica.cc:330] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: stopping tablet replica
I20260619 02:35:02.286077 16017 raft_consensus.cc:2229] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:35:02.286420 16017 raft_consensus.cc:2258] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:02.286811 16019 tablet_replica.cc:330] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: stopping tablet replica
I20260619 02:35:02.286755 15481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:35:02.287118 16019 raft_consensus.cc:2229] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:35:02.287396 16019 raft_consensus.cc:2258] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:35:02.287602 15481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:35:02.287684 16018 tablet_replica.cc:330] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: stopping tablet replica
I20260619 02:35:02.287952 16018 raft_consensus.cc:2229] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:35:02.288213 16017 ts_tablet_manager.cc:1905] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:02.288337 16018 raft_consensus.cc:2258] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:02.291147 16017 ts_tablet_manager.cc:1918] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:35:02.291255 16017 log.cc:1196] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/29d2d8a95d7b43d2a8f1d2e5b3663f2c
I20260619 02:35:02.291368 16019 ts_tablet_manager.cc:1905] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:02.291597 16017 ts_tablet_manager.cc:1939] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P 8dd4b6062261428ab17c019f64d3dc09: Deleting consensus metadata
I20260619 02:35:02.293008 15467 catalog_manager.cc:4841] TS 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487): tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c (table test-table [id=41350ff8123d4e53aa860e8b26d7e3d1]) successfully deleted
I20260619 02:35:02.294063 16019 ts_tablet_manager.cc:1918] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:35:02.294157 16019 log.cc:1196] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/29d2d8a95d7b43d2a8f1d2e5b3663f2c
I20260619 02:35:02.294440 16019 ts_tablet_manager.cc:1939] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P c19da35034964e16bb46cb6ecbc8b948: Deleting consensus metadata
I20260619 02:35:02.294852 15619 tablet_service.cc:1467] Processing CreateTablet for tablet 4533267333eb431b94579f5fe584c06c (DEFAULT_TABLE table=test-table [id=9223fbcd8d1948e3b3c0e4b13479ec1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:02.295130 15619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4533267333eb431b94579f5fe584c06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.295149 15759 tablet_service.cc:1467] Processing CreateTablet for tablet 4533267333eb431b94579f5fe584c06c (DEFAULT_TABLE table=test-table [id=9223fbcd8d1948e3b3c0e4b13479ec1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:02.295425 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4533267333eb431b94579f5fe584c06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.296316 16018 ts_tablet_manager.cc:1905] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:02.297463 15994 tablet_bootstrap.cc:492] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: Bootstrap starting.
I20260619 02:35:02.298081 15993 tablet_bootstrap.cc:492] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: Bootstrap starting.
I20260619 02:35:02.298367 15466 catalog_manager.cc:4841] TS c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111): tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c (table test-table [id=41350ff8123d4e53aa860e8b26d7e3d1]) successfully deleted
I20260619 02:35:02.298516 15994 tablet_bootstrap.cc:654] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.298777 16018 ts_tablet_manager.cc:1918] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:35:02.298866 16018 log.cc:1196] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/29d2d8a95d7b43d2a8f1d2e5b3663f2c
I20260619 02:35:02.299171 15993 tablet_bootstrap.cc:654] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.299188 16018 ts_tablet_manager.cc:1939] T 29d2d8a95d7b43d2a8f1d2e5b3663f2c P e8c600c8d984495c94dedd0d10707961: Deleting consensus metadata
I20260619 02:35:02.299376 15899 tablet_service.cc:1467] Processing CreateTablet for tablet 4533267333eb431b94579f5fe584c06c (DEFAULT_TABLE table=test-table [id=9223fbcd8d1948e3b3c0e4b13479ec1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:02.299651 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4533267333eb431b94579f5fe584c06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.300345 15469 catalog_manager.cc:4841] TS e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437): tablet 29d2d8a95d7b43d2a8f1d2e5b3663f2c (table test-table [id=41350ff8123d4e53aa860e8b26d7e3d1]) successfully deleted
I20260619 02:35:02.300819 15994 tablet_bootstrap.cc:492] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: No bootstrap required, opened a new log
I20260619 02:35:02.300905 15994 ts_tablet_manager.cc:1397] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:35:02.301641 15994 raft_consensus.cc:348] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.301771 15994 raft_consensus.cc:374] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.301920 15995 tablet_bootstrap.cc:492] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: Bootstrap starting.
I20260619 02:35:02.302832 15995 tablet_bootstrap.cc:654] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.301806 15994 raft_consensus.cc:729] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19da35034964e16bb46cb6ecbc8b948, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.303514 15994 consensus_queue.cc:260] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.303815 15994 ts_tablet_manager.cc:1428] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260619 02:35:02.304137 15995 tablet_bootstrap.cc:492] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: No bootstrap required, opened a new log
I20260619 02:35:02.304219 15995 ts_tablet_manager.cc:1397] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:35:02.304559 15993 tablet_bootstrap.cc:492] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: No bootstrap required, opened a new log
I20260619 02:35:02.304637 15993 ts_tablet_manager.cc:1397] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 02:35:02.304713 15995 raft_consensus.cc:348] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.304823 15995 raft_consensus.cc:374] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.304857 15995 raft_consensus.cc:729] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd4b6062261428ab17c019f64d3dc09, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.304991 15995 consensus_queue.cc:260] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.305071 15993 raft_consensus.cc:348] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.305171 15993 raft_consensus.cc:374] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.305200 15993 raft_consensus.cc:729] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c600c8d984495c94dedd0d10707961, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.305238 15995 ts_tablet_manager.cc:1428] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:35:02.305312 15993 consensus_queue.cc:260] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.305570 15993 ts_tablet_manager.cc:1428] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:35:02.324064 16001 raft_consensus.cc:482] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:35:02.324232 16001 raft_consensus.cc:504] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.325574 16001 leader_election.cc:288] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437), 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487)
I20260619 02:35:02.331317 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.326289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:51623 (local address 127.7.194.195:37487)
0619 02:35:02.326407 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.326411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.326783 (+ 372us) server_negotiation.cc:407] Connection header received
0619 02:35:02.326951 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.326955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.327009 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.327080 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:02.327444 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.327980 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.329756 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.330390 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.330488 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.330541 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.331086 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.331178 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:35:02.331377 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.325825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:42437 (local address 127.7.194.193:59947)
0619 02:35:02.326541 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.326555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.326705 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.327013 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.327015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.327030 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:02.327192 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.327200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.328266 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.328269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.329609 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.329620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.330666 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.330669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.330894 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.330958 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.331032 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.331172 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":221,"thread_start_us":81,"threads_started":1}
I20260619 02:35:02.331378 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.325903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:37487 (local address 127.7.194.193:51623)
0619 02:35:02.326702 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.326714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.326822 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.327119 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.327124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.327139 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:02.327271 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.327278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.328084 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.328089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.329609 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.329620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.330666 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.330669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.330882 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.330953 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.331032 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.331169 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":216,"thread_start_us":96,"threads_started":1}
I20260619 02:35:02.331796 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.326237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:59947 (local address 127.7.194.194:42437)
0619 02:35:02.326403 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.326408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.326643 (+ 235us) server_negotiation.cc:407] Connection header received
0619 02:35:02.326865 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.326869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.326930 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.326996 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:02.327346 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.328163 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.329757 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.330387 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.330488 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.330541 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.331571 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.331654 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:35:02.332367 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4533267333eb431b94579f5fe584c06c" candidate_uuid: "c19da35034964e16bb46cb6ecbc8b948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8c600c8d984495c94dedd0d10707961" is_pre_election: true
I20260619 02:35:02.332499 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4533267333eb431b94579f5fe584c06c" candidate_uuid: "c19da35034964e16bb46cb6ecbc8b948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd4b6062261428ab17c019f64d3dc09" is_pre_election: true
I20260619 02:35:02.332643 15919 raft_consensus.cc:2454] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19da35034964e16bb46cb6ecbc8b948 in term 0.
I20260619 02:35:02.332670 15779 raft_consensus.cc:2454] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19da35034964e16bb46cb6ecbc8b948 in term 0.
I20260619 02:35:02.333042 15571 leader_election.cc:302] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dd4b6062261428ab17c019f64d3dc09, c19da35034964e16bb46cb6ecbc8b948; no voters:
I20260619 02:35:02.333326 16001 raft_consensus.cc:2781] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:35:02.333413 16001 raft_consensus.cc:482] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:35:02.333454 16001 raft_consensus.cc:3037] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.334573 16001 raft_consensus.cc:504] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.334998 16001 leader_election.cc:288] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [CANDIDATE]: Term 1 election: Requested vote from peers e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437), 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487)
I20260619 02:35:02.335289 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4533267333eb431b94579f5fe584c06c" candidate_uuid: "c19da35034964e16bb46cb6ecbc8b948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8c600c8d984495c94dedd0d10707961"
I20260619 02:35:02.335337 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4533267333eb431b94579f5fe584c06c" candidate_uuid: "c19da35034964e16bb46cb6ecbc8b948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd4b6062261428ab17c019f64d3dc09"
I20260619 02:35:02.335412 15779 raft_consensus.cc:3037] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.335489 15919 raft_consensus.cc:3037] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.336548 15919 raft_consensus.cc:2454] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19da35034964e16bb46cb6ecbc8b948 in term 1.
I20260619 02:35:02.336588 15779 raft_consensus.cc:2454] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19da35034964e16bb46cb6ecbc8b948 in term 1.
I20260619 02:35:02.336845 15573 leader_election.cc:302] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19da35034964e16bb46cb6ecbc8b948, e8c600c8d984495c94dedd0d10707961; no voters:
I20260619 02:35:02.337028 16001 raft_consensus.cc:2781] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:02.337256 16001 raft_consensus.cc:686] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 1 LEADER]: Becoming Leader. State: Replica: c19da35034964e16bb46cb6ecbc8b948, State: Running, Role: LEADER
I20260619 02:35:02.337522 16001 consensus_queue.cc:237] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.340190 15481 catalog_manager.cc:5495] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 reported cstate change: term changed from 0 to 1, leader changed from <none> to c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193). New cstate: current_term: 1 leader_uuid: "c19da35034964e16bb46cb6ecbc8b948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:35:02 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836500, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.254@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:35:02.396775 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.387662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.7.194.254:40457)
0619 02:35:02.387990 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.387994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.388010 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:35:02.388112 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.388114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.388159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.388249 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:02.388907 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.389448 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.390279 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.390501 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.390604 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.393095 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:02.393098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:02.393100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:02.393120 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:02.394849 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:02.395351 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:02.395358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:02.395361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:02.395449 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:02.395741 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:02.395744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:02.395746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:02.395960 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:02.396066 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.396392 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.396589 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":274,"thread_start_us":70,"threads_started":1}
I20260619 02:35:02.396775 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.387629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:40457 (local address 127.0.0.1:58618)
0619 02:35:02.387867 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.387884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.388006 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.388280 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.388283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.388689 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:02.388785 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.388791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.389591 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.389596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.390147 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.390153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.390617 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.390621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.390757 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.391399 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:02.391412 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:02.392949 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:02.394965 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:02.394978 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:02.394986 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:02.395240 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:02.395550 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:02.395553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:02.395555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:02.395644 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:02.396076 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:02.396085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:02.396206 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.396328 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.396526 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":36,"thread_start_us":90,"threads_started":1}
I20260619 02:35:02.401124 15481 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:58618:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:35:02.401925 15481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:35:02.407229 15619 tablet_service.cc:1467] Processing CreateTablet for tablet 468b22f4ab1344a7becc259b9a330343 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4050f728b7134fc5bd020c8de7728056]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:35:02.407351 15759 tablet_service.cc:1467] Processing CreateTablet for tablet 468b22f4ab1344a7becc259b9a330343 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4050f728b7134fc5bd020c8de7728056]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:35:02.407482 15899 tablet_service.cc:1467] Processing CreateTablet for tablet 468b22f4ab1344a7becc259b9a330343 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4050f728b7134fc5bd020c8de7728056]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:35:02.407692 15619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468b22f4ab1344a7becc259b9a330343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.407764 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468b22f4ab1344a7becc259b9a330343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.407824 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468b22f4ab1344a7becc259b9a330343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.410890 15993 tablet_bootstrap.cc:492] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: Bootstrap starting.
I20260619 02:35:02.410955 15995 tablet_bootstrap.cc:492] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: Bootstrap starting.
I20260619 02:35:02.411895 15995 tablet_bootstrap.cc:654] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.411950 15993 tablet_bootstrap.cc:654] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.413228 15995 tablet_bootstrap.cc:492] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: No bootstrap required, opened a new log
I20260619 02:35:02.413324 15995 ts_tablet_manager.cc:1397] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:35:02.413344 15994 tablet_bootstrap.cc:492] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: Bootstrap starting.
I20260619 02:35:02.413910 15995 raft_consensus.cc:348] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.414057 15995 raft_consensus.cc:374] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.414105 15995 raft_consensus.cc:729] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd4b6062261428ab17c019f64d3dc09, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.414299 15994 tablet_bootstrap.cc:654] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.414266 15995 consensus_queue.cc:260] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.414613 15995 ts_tablet_manager.cc:1428] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:35:02.414549 16000 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.414696 16000 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:35:02.415482 15993 tablet_bootstrap.cc:492] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: No bootstrap required, opened a new log
I20260619 02:35:02.415580 15993 ts_tablet_manager.cc:1397] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 02:35:02.416064 15993 raft_consensus.cc:348] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.416195 15993 raft_consensus.cc:374] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.416232 15993 raft_consensus.cc:729] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c600c8d984495c94dedd0d10707961, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.416365 15993 consensus_queue.cc:260] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.416855 15993 ts_tablet_manager.cc:1428] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:35:02.416806 16013 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.416994 16013 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:35:02.417903 15994 tablet_bootstrap.cc:492] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: No bootstrap required, opened a new log
I20260619 02:35:02.418033 15994 ts_tablet_manager.cc:1397] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 02:35:02.418495 15994 raft_consensus.cc:348] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.418637 15994 raft_consensus.cc:374] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.418673 15994 raft_consensus.cc:729] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19da35034964e16bb46cb6ecbc8b948, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.418805 15994 consensus_queue.cc:260] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.419090 15994 ts_tablet_manager.cc:1428] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:35:02.419242 16001 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.419384 16001 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:35:02.449738 16000 raft_consensus.cc:482] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:35:02.449901 16000 raft_consensus.cc:504] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.451215 16000 leader_election.cc:288] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111), e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437)
I20260619 02:35:02.455571 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.451585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:39245 (local address 127.7.194.193:41111)
0619 02:35:02.451708 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.451713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.451949 (+ 236us) server_negotiation.cc:407] Connection header received
0619 02:35:02.452121 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.452129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.452186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.452263 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:02.452515 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.453127 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.454477 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.454884 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.455012 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.455068 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.455330 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.455430 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:35:02.455591 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.451647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:33451 (local address 127.7.194.194:42437)
0619 02:35:02.451820 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.451825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.452069 (+ 244us) server_negotiation.cc:407] Connection header received
0619 02:35:02.452258 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.452261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.452310 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.452380 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:02.452662 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.453409 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.454643 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.455041 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.455172 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.455225 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.455398 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.455486 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 02:35:02.455588 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.451466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41111 (local address 127.7.194.195:39245)
0619 02:35:02.451858 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.451873 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.451999 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.452272 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.452275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.452287 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:02.452408 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.452413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.453275 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.453278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.454352 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.454361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.455012 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.455015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.455116 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.455134 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.455293 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.455389 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":91,"threads_started":1}
I20260619 02:35:02.455595 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.451466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:42437 (local address 127.7.194.195:33451)
0619 02:35:02.451988 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.452002 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.452131 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.452400 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.452403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.452419 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:02.452544 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.452550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.453581 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.453584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.454512 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.454520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.455159 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.455163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.455269 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.455298 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.455374 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.455437 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":242,"thread_start_us":68,"threads_started":1}
I20260619 02:35:02.456314 15639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468b22f4ab1344a7becc259b9a330343" candidate_uuid: "8dd4b6062261428ab17c019f64d3dc09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19da35034964e16bb46cb6ecbc8b948" is_pre_election: true
I20260619 02:35:02.456295 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468b22f4ab1344a7becc259b9a330343" candidate_uuid: "8dd4b6062261428ab17c019f64d3dc09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8c600c8d984495c94dedd0d10707961" is_pre_election: true
I20260619 02:35:02.456472 15779 raft_consensus.cc:2454] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dd4b6062261428ab17c019f64d3dc09 in term 0.
I20260619 02:35:02.456472 15639 raft_consensus.cc:2454] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dd4b6062261428ab17c019f64d3dc09 in term 0.
I20260619 02:35:02.456823 15850 leader_election.cc:302] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dd4b6062261428ab17c019f64d3dc09, c19da35034964e16bb46cb6ecbc8b948; no voters:
I20260619 02:35:02.457130 16000 raft_consensus.cc:2781] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:35:02.457192 16000 raft_consensus.cc:482] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:35:02.457233 16000 raft_consensus.cc:3037] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.458310 16000 raft_consensus.cc:504] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.458670 16000 leader_election.cc:288] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [CANDIDATE]: Term 1 election: Requested vote from peers c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111), e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437)
I20260619 02:35:02.459079 15639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468b22f4ab1344a7becc259b9a330343" candidate_uuid: "8dd4b6062261428ab17c019f64d3dc09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19da35034964e16bb46cb6ecbc8b948"
I20260619 02:35:02.459177 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468b22f4ab1344a7becc259b9a330343" candidate_uuid: "8dd4b6062261428ab17c019f64d3dc09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8c600c8d984495c94dedd0d10707961"
I20260619 02:35:02.459219 15639 raft_consensus.cc:3037] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.459292 15779 raft_consensus.cc:3037] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.460388 15779 raft_consensus.cc:2454] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dd4b6062261428ab17c019f64d3dc09 in term 1.
I20260619 02:35:02.460388 15639 raft_consensus.cc:2454] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dd4b6062261428ab17c019f64d3dc09 in term 1.
I20260619 02:35:02.460705 15853 leader_election.cc:302] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dd4b6062261428ab17c019f64d3dc09, e8c600c8d984495c94dedd0d10707961; no voters:
I20260619 02:35:02.460901 16000 raft_consensus.cc:2781] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:02.461112 16000 raft_consensus.cc:686] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [term 1 LEADER]: Becoming Leader. State: Replica: 8dd4b6062261428ab17c019f64d3dc09, State: Running, Role: LEADER
I20260619 02:35:02.461385 16000 consensus_queue.cc:237] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.462635 16039 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: TxnStatusTablet state changed. Reason: New leader 8dd4b6062261428ab17c019f64d3dc09. Latest consensus state: current_term: 1 leader_uuid: "8dd4b6062261428ab17c019f64d3dc09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.462767 16039 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:35:02.463058 16041 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:35:02.463228 16041 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:35:02.464013 15480 catalog_manager.cc:5495] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195). New cstate: current_term: 1 leader_uuid: "8dd4b6062261428ab17c019f64d3dc09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } health_report { overall_health: HEALTHY } } }
I20260619 02:35:02.488318 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.483826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.7.194.195:37487)
0619 02:35:02.483955 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.483961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.483977 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:35:02.484193 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.484197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.484267 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.484397 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:35:02.484740 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.485334 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.486741 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.486955 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.487041 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.487408 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:35:02.487890 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:35:02.487976 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.488133 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.488190 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:35:02.488371 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.483729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:37487 (local address 127.0.0.1:35628)
0619 02:35:02.483894 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.483913 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.484040 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.484446 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.484450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.484461 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:35:02.484596 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.484603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.485444 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.485447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.486607 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.486619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.487078 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.487082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.487191 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.487253 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:35:02.487994 (+ 741us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:35:02.488010 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.488094 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.488202 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 02:35:02.491839 15779 raft_consensus.cc:1261] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Refusing update from remote peer 8dd4b6062261428ab17c019f64d3dc09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:02.491863 15639 raft_consensus.cc:1261] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Refusing update from remote peer 8dd4b6062261428ab17c019f64d3dc09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:02.492553 16000 consensus_queue.cc:1035] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:02.492774 16039 consensus_queue.cc:1035] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:02.494455 16001 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: TxnStatusTablet state changed. Reason: New leader 8dd4b6062261428ab17c019f64d3dc09. Latest consensus state: current_term: 1 leader_uuid: "8dd4b6062261428ab17c019f64d3dc09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.494585 16001 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:35:02.496254 16013 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: TxnStatusTablet state changed. Reason: New leader 8dd4b6062261428ab17c019f64d3dc09. Latest consensus state: current_term: 1 leader_uuid: "8dd4b6062261428ab17c019f64d3dc09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.496380 16013 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:35:02.496557 16039 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dd4b6062261428ab17c019f64d3dc09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.496662 16039 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:35:02.498436 16039 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dd4b6062261428ab17c019f64d3dc09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.498570 16039 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P 8dd4b6062261428ab17c019f64d3dc09: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:35:02.499409 16013 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dd4b6062261428ab17c019f64d3dc09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.499526 16013 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P e8c600c8d984495c94dedd0d10707961: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:35:02.499418 16007 mvcc.cc:205] Tried to move back new op lower bound from 7298402314200420352 to 7298402314084478976. Current Snapshot: MvccSnapshot[applied={T|T < 7298402314200420352}]
I20260619 02:35:02.500735 16001 tablet_replica.cc:439] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dd4b6062261428ab17c019f64d3dc09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } } }
I20260619 02:35:02.500859 16001 tablet_replica.cc:442] T 468b22f4ab1344a7becc259b9a330343 P c19da35034964e16bb46cb6ecbc8b948: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:35:02.509160 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.504687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.7.194.193:41111)
0619 02:35:02.504833 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.504838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.504853 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:35:02.504940 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.504943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.505009 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.505126 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:35:02.505348 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.506237 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.507133 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.507538 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.507652 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.507950 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:35:02.508614 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:35:02.508746 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.508926 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.508998 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:35:02.520810 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.517451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:43431 (local address 127.7.194.195:37487)
0619 02:35:02.517714 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.517722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.517778 (+ 56us) server_negotiation.cc:407] Connection header received
0619 02:35:02.517935 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.517938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.517988 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.518066 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:35:02.518318 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.518828 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.519718 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.519935 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.520041 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.520342 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:35:02.520440 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:35:02.520512 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.520606 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.520652 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260619 02:35:02.520828 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.517343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:37487 (local address 127.7.194.193:43431)
0619 02:35:02.517679 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.517695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.517838 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.518076 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.518079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.518085 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:35:02.518215 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.518220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.518934 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.518939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.519593 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.519601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.520041 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.520045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.520141 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.520197 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:35:02.520528 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:35:02.520535 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.520595 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.520673 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":118,"threads_started":1}
I20260619 02:35:02.528061 15919 raft_consensus.cc:1261] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Refusing update from remote peer c19da35034964e16bb46cb6ecbc8b948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:02.528772 15779 raft_consensus.cc:1261] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Refusing update from remote peer c19da35034964e16bb46cb6ecbc8b948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:02.528964 16001 consensus_queue.cc:1035] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:02.529690 16001 consensus_queue.cc:1035] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:02.554085 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.550279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41111 (local address 127.7.194.195:46555)
0619 02:35:02.550612 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:02.550625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:02.550721 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:02.550967 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:02.550970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:02.550977 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:35:02.551115 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.551122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.551851 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.551856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.552717 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:02.552726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.553291 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.553294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:02.553393 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.553437 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:35:02.553772 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:35:02.553780 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:35:02.553841 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 02:35:02.553928 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":136,"threads_started":1}
I20260619 02:35:02.554066 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.550320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:46555 (local address 127.7.194.193:41111)
0619 02:35:02.550442 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.550447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.550692 (+ 245us) server_negotiation.cc:407] Connection header received
0619 02:35:02.550794 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.550798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.550876 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.550958 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:35:02.551235 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.551746 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.552844 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.553157 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.553282 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.553597 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:35:02.553679 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:35:02.553758 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.553861 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.553915 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:35:02.578274 15480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:02.578454 15480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:02.581103 15480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: Sending DeleteTablet for 3 replicas of tablet 4533267333eb431b94579f5fe584c06c
I20260619 02:35:02.582486 15759 tablet_service.cc:1514] Processing DeleteTablet for tablet 4533267333eb431b94579f5fe584c06c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:02 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:38440
I20260619 02:35:02.582727 16018 tablet_replica.cc:330] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: stopping tablet replica
I20260619 02:35:02.582916 16018 raft_consensus.cc:2229] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:35:02.583060 16018 raft_consensus.cc:2258] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:02.584005 16018 ts_tablet_manager.cc:1905] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:02.585929 15619 tablet_service.cc:1514] Processing DeleteTablet for tablet 4533267333eb431b94579f5fe584c06c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:02 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:37000
I20260619 02:35:02.586290 16019 tablet_replica.cc:330] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: stopping tablet replica
I20260619 02:35:02.586421 16019 raft_consensus.cc:2229] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:35:02.586756 16019 raft_consensus.cc:2258] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:02.586869 15899 tablet_service.cc:1514] Processing DeleteTablet for tablet 4533267333eb431b94579f5fe584c06c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:02 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:35618
I20260619 02:35:02.587641 16019 ts_tablet_manager.cc:1905] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:02.587738 16017 tablet_replica.cc:330] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: stopping tablet replica
I20260619 02:35:02.587877 16017 raft_consensus.cc:2229] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:35:02.588006 16017 raft_consensus.cc:2258] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:02.589622 16017 ts_tablet_manager.cc:1905] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:02.590970 16019 ts_tablet_manager.cc:1918] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:35:02.591058 16019 log.cc:1196] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/4533267333eb431b94579f5fe584c06c
I20260619 02:35:02.591347 16019 ts_tablet_manager.cc:1939] T 4533267333eb431b94579f5fe584c06c P c19da35034964e16bb46cb6ecbc8b948: Deleting consensus metadata
I20260619 02:35:02.592376 15466 catalog_manager.cc:4841] TS c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111): tablet 4533267333eb431b94579f5fe584c06c (table test-table [id=9223fbcd8d1948e3b3c0e4b13479ec1a]) successfully deleted
I20260619 02:35:02.592626 16017 ts_tablet_manager.cc:1918] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:35:02.592733 16017 log.cc:1196] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/4533267333eb431b94579f5fe584c06c
I20260619 02:35:02.592998 16017 ts_tablet_manager.cc:1939] T 4533267333eb431b94579f5fe584c06c P 8dd4b6062261428ab17c019f64d3dc09: Deleting consensus metadata
I20260619 02:35:02.593729 15467 catalog_manager.cc:4841] TS 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487): tablet 4533267333eb431b94579f5fe584c06c (table test-table [id=9223fbcd8d1948e3b3c0e4b13479ec1a]) successfully deleted
I20260619 02:35:02.594301 16018 ts_tablet_manager.cc:1918] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:35:02.594388 16018 log.cc:1196] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/4533267333eb431b94579f5fe584c06c
I20260619 02:35:02.594645 16018 ts_tablet_manager.cc:1939] T 4533267333eb431b94579f5fe584c06c P e8c600c8d984495c94dedd0d10707961: Deleting consensus metadata
I20260619 02:35:02.595337 15469 catalog_manager.cc:4841] TS e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437): tablet 4533267333eb431b94579f5fe584c06c (table test-table [id=9223fbcd8d1948e3b3c0e4b13479ec1a]) successfully deleted
Jun 19 02:35:02 dist-test-slave-9tkz krb5kdc[15437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:35:02 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:02 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:35:02.630585 16033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:02.623604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.7.194.254:40457)
0619 02:35:02.623750 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.623755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.624016 (+ 261us) server_negotiation.cc:407] Connection header received
0619 02:35:02.624182 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.624186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.624244 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.624335 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:02.625095 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.625746 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.626727 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.627064 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.627182 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.630474 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58632: BlockingRecv error: recv got EOF from 127.0.0.1:58632 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 02:35:02 dist-test-slave-9tkz krb5kdc[15437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM
I20260619 02:35:02.641968 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.632749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.7.194.254:40457)
0619 02:35:02.632866 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.632871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.633215 (+ 344us) server_negotiation.cc:407] Connection header received
0619 02:35:02.633357 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.633360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.633413 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.633501 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:02.634137 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.634659 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.635472 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.635716 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.635825 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.638625 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:02.638629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:02.638632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:02.638667 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:02.640177 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:02.640605 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:02.640609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:02.640611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:02.640703 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:02.640973 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:02.640977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:02.640979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:02.641170 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:02.641311 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.641601 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.641755 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:35:02.644654 15480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:35:02.645146 15480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:35:02.650094 15619 tablet_service.cc:1467] Processing CreateTablet for tablet c664e531e2f8469f969d7820ccfa6eec (DEFAULT_TABLE table=test-table [id=d021dd4d66024595afb18264e4487b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:02.650446 15619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c664e531e2f8469f969d7820ccfa6eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.650403 15899 tablet_service.cc:1467] Processing CreateTablet for tablet c664e531e2f8469f969d7820ccfa6eec (DEFAULT_TABLE table=test-table [id=d021dd4d66024595afb18264e4487b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:02.650658 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c664e531e2f8469f969d7820ccfa6eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.653059 15995 tablet_bootstrap.cc:492] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: Bootstrap starting.
I20260619 02:35:02.653211 15994 tablet_bootstrap.cc:492] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: Bootstrap starting.
I20260619 02:35:02.654184 15994 tablet_bootstrap.cc:654] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.654778 15995 tablet_bootstrap.cc:654] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.654978 15759 tablet_service.cc:1467] Processing CreateTablet for tablet c664e531e2f8469f969d7820ccfa6eec (DEFAULT_TABLE table=test-table [id=d021dd4d66024595afb18264e4487b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:02.655292 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c664e531e2f8469f969d7820ccfa6eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.656126 15995 tablet_bootstrap.cc:492] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: No bootstrap required, opened a new log
I20260619 02:35:02.656208 15995 ts_tablet_manager.cc:1397] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:35:02.656697 15995 raft_consensus.cc:348] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.656841 15995 raft_consensus.cc:374] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.656881 15995 raft_consensus.cc:729] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd4b6062261428ab17c019f64d3dc09, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.656998 15995 consensus_queue.cc:260] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.657251 15995 ts_tablet_manager.cc:1428] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:35:02.658104 15993 tablet_bootstrap.cc:492] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: Bootstrap starting.
I20260619 02:35:02.658808 15993 tablet_bootstrap.cc:654] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.659832 15993 tablet_bootstrap.cc:492] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: No bootstrap required, opened a new log
I20260619 02:35:02.659911 15993 ts_tablet_manager.cc:1397] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:35:02.660319 15993 raft_consensus.cc:348] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.660434 15993 raft_consensus.cc:374] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.660471 15993 raft_consensus.cc:729] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c600c8d984495c94dedd0d10707961, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.660591 15993 consensus_queue.cc:260] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.660868 15993 ts_tablet_manager.cc:1428] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:35:02.664551 15994 tablet_bootstrap.cc:492] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: No bootstrap required, opened a new log
I20260619 02:35:02.664685 15994 ts_tablet_manager.cc:1397] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260619 02:35:02.665127 15994 raft_consensus.cc:348] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.665252 15994 raft_consensus.cc:374] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.665293 15994 raft_consensus.cc:729] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19da35034964e16bb46cb6ecbc8b948, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.665429 15994 consensus_queue.cc:260] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.665795 15994 ts_tablet_manager.cc:1428] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:35:02.880066 16013 raft_consensus.cc:482] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:35:02.880223 16013 raft_consensus.cc:504] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.880692 16013 leader_election.cc:288] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111), 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487)
I20260619 02:35:02.881227 15639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c664e531e2f8469f969d7820ccfa6eec" candidate_uuid: "e8c600c8d984495c94dedd0d10707961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19da35034964e16bb46cb6ecbc8b948" is_pre_election: true
I20260619 02:35:02.881232 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c664e531e2f8469f969d7820ccfa6eec" candidate_uuid: "e8c600c8d984495c94dedd0d10707961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd4b6062261428ab17c019f64d3dc09" is_pre_election: true
I20260619 02:35:02.881404 15639 raft_consensus.cc:2454] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8c600c8d984495c94dedd0d10707961 in term 0.
I20260619 02:35:02.881418 15919 raft_consensus.cc:2454] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8c600c8d984495c94dedd0d10707961 in term 0.
I20260619 02:35:02.881745 15710 leader_election.cc:302] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19da35034964e16bb46cb6ecbc8b948, e8c600c8d984495c94dedd0d10707961; no voters:
I20260619 02:35:02.881913 16013 raft_consensus.cc:2781] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:35:02.881976 16013 raft_consensus.cc:482] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:35:02.882018 16013 raft_consensus.cc:3037] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.883136 16013 raft_consensus.cc:504] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.883504 16013 leader_election.cc:288] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [CANDIDATE]: Term 1 election: Requested vote from peers c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111), 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487)
I20260619 02:35:02.883919 15639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c664e531e2f8469f969d7820ccfa6eec" candidate_uuid: "e8c600c8d984495c94dedd0d10707961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19da35034964e16bb46cb6ecbc8b948"
I20260619 02:35:02.884011 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c664e531e2f8469f969d7820ccfa6eec" candidate_uuid: "e8c600c8d984495c94dedd0d10707961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd4b6062261428ab17c019f64d3dc09"
I20260619 02:35:02.884069 15639 raft_consensus.cc:3037] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.884166 15919 raft_consensus.cc:3037] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:02.885373 15639 raft_consensus.cc:2454] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8c600c8d984495c94dedd0d10707961 in term 1.
I20260619 02:35:02.885373 15919 raft_consensus.cc:2454] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8c600c8d984495c94dedd0d10707961 in term 1.
I20260619 02:35:02.885696 15710 leader_election.cc:302] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19da35034964e16bb46cb6ecbc8b948, e8c600c8d984495c94dedd0d10707961; no voters:
I20260619 02:35:02.885887 16013 raft_consensus.cc:2781] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:02.886082 16013 raft_consensus.cc:686] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 1 LEADER]: Becoming Leader. State: Replica: e8c600c8d984495c94dedd0d10707961, State: Running, Role: LEADER
I20260619 02:35:02.886243 16013 consensus_queue.cc:237] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.887904 15482 catalog_manager.cc:5495] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8c600c8d984495c94dedd0d10707961 (127.7.194.194). New cstate: current_term: 1 leader_uuid: "e8c600c8d984495c94dedd0d10707961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } health_report { overall_health: UNKNOWN } } }
I20260619 02:35:02.906349 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:02.903205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.7.194.194:42437)
0619 02:35:02.903343 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:02.903347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:02.903361 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:35:02.903445 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:02.903447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:02.903490 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:02.903654 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:35:02.903922 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.904432 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.905250 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:02.905467 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:02.905577 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:02.905877 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:35:02.905973 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:35:02.906031 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:02.906111 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 02:35:02.906159 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:35:02.908411 15919 raft_consensus.cc:1261] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Refusing update from remote peer e8c600c8d984495c94dedd0d10707961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:02.908427 15639 raft_consensus.cc:1261] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Refusing update from remote peer e8c600c8d984495c94dedd0d10707961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:02.908839 16013 consensus_queue.cc:1035] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:02.908995 16072 consensus_queue.cc:1035] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:02.916960 15482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:02.917091 15482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:02.918900 15482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: Sending DeleteTablet for 3 replicas of tablet c664e531e2f8469f969d7820ccfa6eec
I20260619 02:35:02.919402 15619 tablet_service.cc:1514] Processing DeleteTablet for tablet c664e531e2f8469f969d7820ccfa6eec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:02 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:37000
I20260619 02:35:02.919531 15899 tablet_service.cc:1514] Processing DeleteTablet for tablet c664e531e2f8469f969d7820ccfa6eec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:02 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:35618
I20260619 02:35:02.919625 16019 tablet_replica.cc:330] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: stopping tablet replica
I20260619 02:35:02.919620 15759 tablet_service.cc:1514] Processing DeleteTablet for tablet c664e531e2f8469f969d7820ccfa6eec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:02 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:38440
I20260619 02:35:02.919713 16017 tablet_replica.cc:330] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: stopping tablet replica
I20260619 02:35:02.919772 16018 tablet_replica.cc:330] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: stopping tablet replica
I20260619 02:35:02.919756 16019 raft_consensus.cc:2229] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:35:02.919829 16017 raft_consensus.cc:2229] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:35:02.919866 16018 raft_consensus.cc:2229] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:35:02.919884 16019 raft_consensus.cc:2258] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:02.919953 16017 raft_consensus.cc:2258] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:02.920039 16018 raft_consensus.cc:2258] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:02.920416 15482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:35:02.920754 16017 ts_tablet_manager.cc:1905] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:02.920855 16018 ts_tablet_manager.cc:1905] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:35:02.920876 15482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:35:02.923287 16018 ts_tablet_manager.cc:1918] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:35:02.923287 16017 ts_tablet_manager.cc:1918] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:35:02.923372 16017 log.cc:1196] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/c664e531e2f8469f969d7820ccfa6eec
I20260619 02:35:02.923372 16018 log.cc:1196] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/c664e531e2f8469f969d7820ccfa6eec
I20260619 02:35:02.923632 16018 ts_tablet_manager.cc:1939] T c664e531e2f8469f969d7820ccfa6eec P e8c600c8d984495c94dedd0d10707961: Deleting consensus metadata
I20260619 02:35:02.923632 16017 ts_tablet_manager.cc:1939] T c664e531e2f8469f969d7820ccfa6eec P 8dd4b6062261428ab17c019f64d3dc09: Deleting consensus metadata
I20260619 02:35:02.924329 16019 ts_tablet_manager.cc:1905] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:02.924460 15467 catalog_manager.cc:4841] TS 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487): tablet c664e531e2f8469f969d7820ccfa6eec (table test-table [id=d021dd4d66024595afb18264e4487b64]) successfully deleted
I20260619 02:35:02.924649 15469 catalog_manager.cc:4841] TS e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437): tablet c664e531e2f8469f969d7820ccfa6eec (table test-table [id=d021dd4d66024595afb18264e4487b64]) successfully deleted
I20260619 02:35:02.926877 16019 ts_tablet_manager.cc:1918] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:35:02.926963 16019 log.cc:1196] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/c664e531e2f8469f969d7820ccfa6eec
I20260619 02:35:02.927284 15619 tablet_service.cc:1467] Processing CreateTablet for tablet dcbb7e3cf77749198ce1324b35706ad1 (DEFAULT_TABLE table=test-table [id=9cd00351eee84585b312b9c374f27191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:02.927415 15759 tablet_service.cc:1467] Processing CreateTablet for tablet dcbb7e3cf77749198ce1324b35706ad1 (DEFAULT_TABLE table=test-table [id=9cd00351eee84585b312b9c374f27191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:02.927537 15619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbb7e3cf77749198ce1324b35706ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.927657 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbb7e3cf77749198ce1324b35706ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.927600 15899 tablet_service.cc:1467] Processing CreateTablet for tablet dcbb7e3cf77749198ce1324b35706ad1 (DEFAULT_TABLE table=test-table [id=9cd00351eee84585b312b9c374f27191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:02.927846 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbb7e3cf77749198ce1324b35706ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:02.929960 15993 tablet_bootstrap.cc:492] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: Bootstrap starting.
I20260619 02:35:02.930068 15994 tablet_bootstrap.cc:492] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948: Bootstrap starting.
I20260619 02:35:02.930158 15995 tablet_bootstrap.cc:492] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: Bootstrap starting.
I20260619 02:35:02.930815 15993 tablet_bootstrap.cc:654] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.930918 15994 tablet_bootstrap.cc:654] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.930918 15995 tablet_bootstrap.cc:654] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:02.931610 16019 ts_tablet_manager.cc:1939] T c664e531e2f8469f969d7820ccfa6eec P c19da35034964e16bb46cb6ecbc8b948: Deleting consensus metadata
I20260619 02:35:02.932268 15466 catalog_manager.cc:4841] TS c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111): tablet c664e531e2f8469f969d7820ccfa6eec (table test-table [id=d021dd4d66024595afb18264e4487b64]) successfully deleted
I20260619 02:35:02.932439 15995 tablet_bootstrap.cc:492] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: No bootstrap required, opened a new log
I20260619 02:35:02.932531 15995 ts_tablet_manager.cc:1397] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 02:35:02.932560 15994 tablet_bootstrap.cc:492] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948: No bootstrap required, opened a new log
I20260619 02:35:02.932626 15994 ts_tablet_manager.cc:1397] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:35:02.932981 15995 raft_consensus.cc:348] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.933022 15994 raft_consensus.cc:348] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.933118 15994 raft_consensus.cc:374] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.933118 15995 raft_consensus.cc:374] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.933152 15995 raft_consensus.cc:729] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd4b6062261428ab17c019f64d3dc09, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.933152 15994 raft_consensus.cc:729] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19da35034964e16bb46cb6ecbc8b948, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.933287 15995 consensus_queue.cc:260] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.933287 15994 consensus_queue.cc:260] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.933590 15995 ts_tablet_manager.cc:1428] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 02:35:02.933625 15994 ts_tablet_manager.cc:1428] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:35:02.934866 15993 tablet_bootstrap.cc:492] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: No bootstrap required, opened a new log
I20260619 02:35:02.934994 15993 ts_tablet_manager.cc:1397] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 02:35:02.935416 15993 raft_consensus.cc:348] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.935544 15993 raft_consensus.cc:374] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:02.935581 15993 raft_consensus.cc:729] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c600c8d984495c94dedd0d10707961, State: Initialized, Role: FOLLOWER
I20260619 02:35:02.935715 15993 consensus_queue.cc:260] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:02.935998 15993 ts_tablet_manager.cc:1428] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:35:03.054802 16001 raft_consensus.cc:482] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:35:03.054950 16001 raft_consensus.cc:504] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:03.055375 16001 leader_election.cc:288] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437), 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487)
I20260619 02:35:03.055836 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbb7e3cf77749198ce1324b35706ad1" candidate_uuid: "c19da35034964e16bb46cb6ecbc8b948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd4b6062261428ab17c019f64d3dc09" is_pre_election: true
I20260619 02:35:03.055938 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbb7e3cf77749198ce1324b35706ad1" candidate_uuid: "c19da35034964e16bb46cb6ecbc8b948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8c600c8d984495c94dedd0d10707961" is_pre_election: true
I20260619 02:35:03.056007 15919 raft_consensus.cc:2454] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19da35034964e16bb46cb6ecbc8b948 in term 0.
I20260619 02:35:03.056084 15779 raft_consensus.cc:2454] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19da35034964e16bb46cb6ecbc8b948 in term 0.
I20260619 02:35:03.056365 15573 leader_election.cc:302] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19da35034964e16bb46cb6ecbc8b948, e8c600c8d984495c94dedd0d10707961; no voters:
I20260619 02:35:03.056633 16001 raft_consensus.cc:2781] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:35:03.056694 16001 raft_consensus.cc:482] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:35:03.056746 16001 raft_consensus.cc:3037] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:03.057832 16001 raft_consensus.cc:504] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:03.058192 16001 leader_election.cc:288] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [CANDIDATE]: Term 1 election: Requested vote from peers e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437), 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487)
I20260619 02:35:03.058526 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbb7e3cf77749198ce1324b35706ad1" candidate_uuid: "c19da35034964e16bb46cb6ecbc8b948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd4b6062261428ab17c019f64d3dc09"
I20260619 02:35:03.058646 15919 raft_consensus.cc:3037] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:03.058710 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbb7e3cf77749198ce1324b35706ad1" candidate_uuid: "c19da35034964e16bb46cb6ecbc8b948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8c600c8d984495c94dedd0d10707961"
I20260619 02:35:03.058822 15779 raft_consensus.cc:3037] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:03.059505 15919 raft_consensus.cc:2454] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19da35034964e16bb46cb6ecbc8b948 in term 1.
I20260619 02:35:03.059762 15571 leader_election.cc:302] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dd4b6062261428ab17c019f64d3dc09, c19da35034964e16bb46cb6ecbc8b948; no voters:
I20260619 02:35:03.059911 16001 raft_consensus.cc:2781] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:03.059938 15779 raft_consensus.cc:2454] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19da35034964e16bb46cb6ecbc8b948 in term 1.
I20260619 02:35:03.060002 16001 raft_consensus.cc:686] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 1 LEADER]: Becoming Leader. State: Replica: c19da35034964e16bb46cb6ecbc8b948, State: Running, Role: LEADER
I20260619 02:35:03.060161 16001 consensus_queue.cc:237] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } }
I20260619 02:35:03.061884 15482 catalog_manager.cc:5495] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 reported cstate change: term changed from 0 to 1, leader changed from <none> to c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193). New cstate: current_term: 1 leader_uuid: "c19da35034964e16bb46cb6ecbc8b948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19da35034964e16bb46cb6ecbc8b948" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 } health_report { overall_health: UNKNOWN } } }
I20260619 02:35:03.096963 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:03.093725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.7.194.193:41111)
0619 02:35:03.094036 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:03.094041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:03.094055 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:35:03.094101 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:03.094104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:03.094159 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:03.094256 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:35:03.094493 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:03.095040 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:03.095889 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:03.096132 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:03.096246 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:03.096489 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:35:03.096570 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:35:03.096647 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:03.096751 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 02:35:03.096804 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":119,"threads_started":1}
I20260619 02:35:03.103279 15919 raft_consensus.cc:1261] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Refusing update from remote peer c19da35034964e16bb46cb6ecbc8b948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:03.103343 15779 raft_consensus.cc:1261] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Refusing update from remote peer c19da35034964e16bb46cb6ecbc8b948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:03.103691 16051 consensus_queue.cc:1035] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c600c8d984495c94dedd0d10707961" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:03.103879 16001 consensus_queue.cc:1035] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd4b6062261428ab17c019f64d3dc09" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:03.137415 15481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:03.137629 15481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:03.139621 15481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a37c65704194797a7c19a36de2b6fe1: Sending DeleteTablet for 3 replicas of tablet dcbb7e3cf77749198ce1324b35706ad1
I20260619 02:35:03.140239 15759 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbb7e3cf77749198ce1324b35706ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:03 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:38440
I20260619 02:35:03.140257 15619 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbb7e3cf77749198ce1324b35706ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:03 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:37000
I20260619 02:35:03.140407 15899 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbb7e3cf77749198ce1324b35706ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:03 UTC) from {username='oryx', principal='oryx/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:35618
I20260619 02:35:03.140436 16018 tablet_replica.cc:330] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: stopping tablet replica
I20260619 02:35:03.140481 16019 tablet_replica.cc:330] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948: stopping tablet replica
I20260619 02:35:03.140568 16018 raft_consensus.cc:2229] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:35:03.140581 16019 raft_consensus.cc:2229] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:35:03.140590 16017 tablet_replica.cc:330] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: stopping tablet replica
I20260619 02:35:03.140717 16018 raft_consensus.cc:2258] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:03.140776 16019 raft_consensus.cc:2258] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:03.140769 16017 raft_consensus.cc:2229] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:35:03.140901 16017 raft_consensus.cc:2258] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:03.141470 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 15554
I20260619 02:35:03.141716 16019 ts_tablet_manager.cc:1905] T dcbb7e3cf77749198ce1324b35706ad1 P c19da35034964e16bb46cb6ecbc8b948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:03.141945 16017 ts_tablet_manager.cc:1905] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:03.143410 16018 ts_tablet_manager.cc:1905] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:03.145318 16017 ts_tablet_manager.cc:1918] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:35:03.145411 16017 log.cc:1196] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/dcbb7e3cf77749198ce1324b35706ad1
I20260619 02:35:03.145524 16018 ts_tablet_manager.cc:1918] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:35:03.145666 16018 log.cc:1196] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/dcbb7e3cf77749198ce1324b35706ad1
I20260619 02:35:03.145723 16017 ts_tablet_manager.cc:1939] T dcbb7e3cf77749198ce1324b35706ad1 P 8dd4b6062261428ab17c019f64d3dc09: Deleting consensus metadata
I20260619 02:35:03.145941 16018 ts_tablet_manager.cc:1939] T dcbb7e3cf77749198ce1324b35706ad1 P e8c600c8d984495c94dedd0d10707961: Deleting consensus metadata
I20260619 02:35:03.146562 15467 catalog_manager.cc:4841] TS 8dd4b6062261428ab17c019f64d3dc09 (127.7.194.195:37487): tablet dcbb7e3cf77749198ce1324b35706ad1 (table test-table [id=9cd00351eee84585b312b9c374f27191]) successfully deleted
I20260619 02:35:03.146584 15469 catalog_manager.cc:4841] TS e8c600c8d984495c94dedd0d10707961 (127.7.194.194:42437): tablet dcbb7e3cf77749198ce1324b35706ad1 (table test-table [id=9cd00351eee84585b312b9c374f27191]) successfully deleted
W20260619 02:35:03.150532 15466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:41111 (error 108)
I20260619 02:35:03.150791 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 15694
I20260619 02:35:03.153223 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:03.152728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41111 (local address 127.0.0.1:37034)
0619 02:35:03.152990 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:03.153079 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":58,"threads_started":1}
W20260619 02:35:03.153386 15466 catalog_manager.cc:4584] TS c19da35034964e16bb46cb6ecbc8b948 (127.7.194.193:41111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcbb7e3cf77749198ce1324b35706ad1: Network error: Client connection negotiation failed: client connection to 127.7.194.193:41111: connect: Connection refused (error 111)
I20260619 02:35:03.158087 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 15834
I20260619 02:35:03.164750 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 15450
2026-06-19T02:35:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3479 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:35:03 dist-test-slave-9tkz krb5kdc[16094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:35:03 dist-test-slave-9tkz krb5kdc[16094](info): set up 2 sockets
Jun 19 02:35:03 dist-test-slave-9tkz krb5kdc[16094](info): commencing operation
krb5kdc: starting...
W20260619 02:35:04.210587 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:35:04 dist-test-slave-9tkz krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:35:04.339123 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44383
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:44383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:04.454738 16107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:04.455034 16107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:04.455119 16107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:04.459301 16107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:35:04.459378 16107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:04.459398 16107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:04.459415 16107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:35:04.459466 16107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:35:04.464942 16107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:44383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:04.466164 16107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:04.467283 16107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:04.476272 16113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:04.476269 16112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:04.476269 16115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:04.476569 16107 server_base.cc:1034] running on GCE node
I20260619 02:35:04.477315 16107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:04.478523 16107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:04.479745 16107 hybrid_clock.cc:648] HybridClock initialized: now 1781836504479729 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T02:35:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1321 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:35:04 dist-test-slave-9tkz krb5kdc[16120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:35:04 dist-test-slave-9tkz krb5kdc[16120](info): set up 2 sockets
Jun 19 02:35:04 dist-test-slave-9tkz krb5kdc[16120](info): commencing operation
krb5kdc: starting...
W20260619 02:35:05.546871 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:35:05 dist-test-slave-9tkz krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:35:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:35:05.685887 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:35519
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:05.802814 16133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:05.803177 16133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:05.803303 16133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:05.807492 16133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:35:05.807613 16133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:05.807662 16133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:05.807701 16133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:35:05.807734 16133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:35:05.813452 16133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:05.814745 16133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:05.815889 16133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:35:05.825081 16133 server_base.cc:1034] running on GCE node
W20260619 02:35:05.825089 16141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:05.825081 16138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:05.825331 16139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:05.825994 16133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:05.827224 16133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:05.828413 16133 hybrid_clock.cc:648] HybridClock initialized: now 1781836505828398 us; error 33 us; skew 500 ppm
Jun 19 02:35:05 dist-test-slave-9tkz krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836505, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:05.831518 16133 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:35:05.832523 16133 webserver.cc:460] Webserver started at http://127.7.194.254:36663/ using document root <none> and password file <none>
I20260619 02:35:05.833137 16133 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:05.833228 16133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:05.833474 16133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:05.835765 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "a690804f9cb0434ea1cf75d8ef2d080e"
format_stamp: "Formatted at 2026-06-19 02:35:05 on dist-test-slave-9tkz"
server_key: "f83b230b1dd409f0081e442f18c518cd"
server_key_iv: "aca615654b1ab9200a7f3b9fbb6de4bd"
server_key_version: "encryptionkey@0"
I20260619 02:35:05.836382 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "a690804f9cb0434ea1cf75d8ef2d080e"
format_stamp: "Formatted at 2026-06-19 02:35:05 on dist-test-slave-9tkz"
server_key: "f83b230b1dd409f0081e442f18c518cd"
server_key_iv: "aca615654b1ab9200a7f3b9fbb6de4bd"
server_key_version: "encryptionkey@0"
I20260619 02:35:05.840554 16133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:35:05.843287 16148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:05.844929 16133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:35:05.845086 16133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "a690804f9cb0434ea1cf75d8ef2d080e"
format_stamp: "Formatted at 2026-06-19 02:35:05 on dist-test-slave-9tkz"
server_key: "f83b230b1dd409f0081e442f18c518cd"
server_key_iv: "aca615654b1ab9200a7f3b9fbb6de4bd"
server_key_version: "encryptionkey@0"
I20260619 02:35:05.845212 16133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:05.861822 16133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:05.865018 16133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:05.865231 16133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:05.872424 16133 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:35519
I20260619 02:35:05.872469 16200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:35519 every 8 connection(s)
I20260619 02:35:05.873466 16133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:35:05.876391 16201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:05.882125 16201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: Bootstrap starting.
I20260619 02:35:05.882545 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 16133
I20260619 02:35:05.882710 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:35:05.882994 7947 external_mini_cluster.cc:1428] Setting key d211092137fe23da22346e0532ef32e7
I20260619 02:35:05.884924 16201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:05.885751 16201 log.cc:824] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:05.887658 16201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: No bootstrap required, opened a new log
I20260619 02:35:05.891595 16201 raft_consensus.cc:348] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } }
I20260619 02:35:05.891861 16201 raft_consensus.cc:374] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:05.891921 16201 raft_consensus.cc:729] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a690804f9cb0434ea1cf75d8ef2d080e, State: Initialized, Role: FOLLOWER
Jun 19 02:35:05 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:35:05.892426 16201 consensus_queue.cc:260] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } }
I20260619 02:35:05.892573 16201 raft_consensus.cc:388] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:35:05.892638 16201 raft_consensus.cc:482] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:35:05.892735 16201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:05.894054 16201 raft_consensus.cc:504] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } }
I20260619 02:35:05.894428 16201 leader_election.cc:302] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a690804f9cb0434ea1cf75d8ef2d080e; no voters:
I20260619 02:35:05.894781 16201 leader_election.cc:288] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:35:05.894912 16206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:05.895138 16206 raft_consensus.cc:686] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 1 LEADER]: Becoming Leader. State: Replica: a690804f9cb0434ea1cf75d8ef2d080e, State: Running, Role: LEADER
I20260619 02:35:05.895545 16206 consensus_queue.cc:237] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } }
I20260619 02:35:05.896090 16201 sys_catalog.cc:564] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:35:05.897312 16207 sys_catalog.cc:455] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } } }
I20260619 02:35:05.897439 16207 sys_catalog.cc:458] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:05.897737 16207 sys_catalog.cc:455] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a690804f9cb0434ea1cf75d8ef2d080e. Latest consensus state: current_term: 1 leader_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } } }
I20260619 02:35:05.897810 16207 sys_catalog.cc:458] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:05.899273 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:05.884424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.7.194.254:35519)
0619 02:35:05.884847 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:05.884863 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:05.884901 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:35:05.885687 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:05.885708 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:05.886104 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:05.886462 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:05.887311 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:05.888367 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:05.889264 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:05.889850 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:05.890021 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:05.892848 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:05.892856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:05.892876 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:05.892913 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:05.895026 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:05.895674 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:05.895680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:05.895687 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:05.895827 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:05.896143 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:05.896147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:05.896149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:05.896683 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:05.897713 (+ 1030us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:05.897876 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 02:35:05.898234 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":139,"threads_started":1}
W20260619 02:35:05.901582 16221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:35:05.901659 16221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:35:05.901878 16216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:35:05.902791 16216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:35:05.908795 16216 catalog_manager.cc:1347] Generated new cluster ID: 0f93abbd0b514812948a6f0ba497b159
I20260619 02:35:05.908891 16216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:35:05.927109 16216 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:35:05.928160 16216 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:35:05.942848 16216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: Generated new TSK 0
I20260619 02:35:05.943667 16216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:35:05.980620 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:35519
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:06.099164 16227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:06.099472 16227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:06.099555 16227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:06.103777 16227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:06.103854 16227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:06.103957 16227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:35:06.110121 16227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:06.111377 16227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:06.112474 16227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:06.122592 16233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:06.122570 16227 server_base.cc:1034] running on GCE node
W20260619 02:35:06.122565 16232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:06.122565 16235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:06.123304 16227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:06.124079 16227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:06.125290 16227 hybrid_clock.cc:648] HybridClock initialized: now 1781836506125271 us; error 32 us; skew 500 ppm
Jun 19 02:35:06 dist-test-slave-9tkz krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836506, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:06.128461 16227 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:35:06.129606 16227 webserver.cc:460] Webserver started at http://127.7.194.193:45843/ using document root <none> and password file <none>
I20260619 02:35:06.130234 16227 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:06.130334 16227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:06.130580 16227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:06.132834 16227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "916f748c74af459da77ba6d121fb210e"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "d654462f7d748b74fa6ca8b6afe78cb6"
server_key_iv: "a7daca2a43f3ad9e66f85ae630934818"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.133448 16227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "916f748c74af459da77ba6d121fb210e"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "d654462f7d748b74fa6ca8b6afe78cb6"
server_key_iv: "a7daca2a43f3ad9e66f85ae630934818"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.137499 16227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:35:06.140136 16242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.141444 16227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:35:06.141631 16227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "916f748c74af459da77ba6d121fb210e"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "d654462f7d748b74fa6ca8b6afe78cb6"
server_key_iv: "a7daca2a43f3ad9e66f85ae630934818"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.141757 16227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:06.161199 16227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:06.164660 16227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:06.164898 16227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:06.165633 16227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:06.166584 16227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:06.166661 16227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.166736 16227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:06.166788 16227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.176749 16227 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:40877
I20260619 02:35:06.176772 16355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:40877 every 8 connection(s)
I20260619 02:35:06.177884 16227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:35:06 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836506, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:35:06.187893 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 16227
I20260619 02:35:06.188042 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:35:06.188324 7947 external_mini_cluster.cc:1428] Setting key fc7e6c05575ea15ed046829c85cda69c
I20260619 02:35:06.192627 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:06.179833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:60475 (local address 127.7.194.254:35519)
0619 02:35:06.179996 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:06.180000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:06.180888 (+ 888us) server_negotiation.cc:407] Connection header received
0619 02:35:06.182315 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:06.182318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:06.182385 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:06.182473 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:06.183705 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.184241 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.185467 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.185769 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.185882 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:06.188607 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:06.188611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:06.188614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:06.188646 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:06.190340 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:06.190882 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:06.190887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:06.190889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:06.190974 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:06.191418 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:06.191422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:06.191423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:06.191675 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:06.191736 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:06.192323 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 02:35:06.192458 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:35:06.193536 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:06.180256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:60475)
0619 02:35:06.180716 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:06.180762 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:06.182077 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:06.182658 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:06.182665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:06.183052 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:06.183556 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:06.183570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.184396 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.184404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:06.185333 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:06.185343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.185930 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.185934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:06.186109 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:06.186715 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:06.186732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:06.188414 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:06.190472 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:06.190478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:06.190496 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:06.190772 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:06.191065 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:06.191069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:06.191071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:06.191221 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:06.191813 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:06.191818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:06.192074 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 02:35:06.192274 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 02:35:06.192632 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
I20260619 02:35:06.194880 16356 heartbeater.cc:347] Connected to a master server at 127.7.194.254:35519
I20260619 02:35:06.195179 16356 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:06.195876 16356 heartbeater.cc:511] Master 127.7.194.254:35519 requested a full tablet report, sending...
I20260619 02:35:06.197811 16165 ts_manager.cc:194] Registered new tserver with Master: 916f748c74af459da77ba6d121fb210e (127.7.194.193:40877)
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:35:06.199258 16165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:60475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:35:06.217350 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:35519
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:06.332083 16361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:06.332383 16361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:06.332471 16361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:06.336674 16361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:06.336752 16361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:06.336853 16361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:35:06.342485 16361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:06.343698 16361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:06.344854 16361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:06.355064 16366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:06.355163 16369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:06.355197 16367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:06.355377 16361 server_base.cc:1034] running on GCE node
I20260619 02:35:06.355808 16361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:06.356508 16361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:06.357672 16361 hybrid_clock.cc:648] HybridClock initialized: now 1781836506357662 us; error 31 us; skew 500 ppm
Jun 19 02:35:06 dist-test-slave-9tkz krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836506, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:06.363615 16361 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:35:06.364581 16361 webserver.cc:460] Webserver started at http://127.7.194.194:36515/ using document root <none> and password file <none>
I20260619 02:35:06.365171 16361 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:06.365257 16361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:06.365491 16361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:06.367832 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "4550c9003dac45cd8e7087b36722565b"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "643ffb95bd4d6c363704639f8e49ebcb"
server_key_iv: "935d7a1f91996a0c8168d34743ea219d"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.368451 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "4550c9003dac45cd8e7087b36722565b"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "643ffb95bd4d6c363704639f8e49ebcb"
server_key_iv: "935d7a1f91996a0c8168d34743ea219d"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.372534 16361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:35:06.375310 16376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.376602 16361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:35:06.376756 16361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "4550c9003dac45cd8e7087b36722565b"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "643ffb95bd4d6c363704639f8e49ebcb"
server_key_iv: "935d7a1f91996a0c8168d34743ea219d"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.376879 16361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:06.394488 16361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:06.398312 16361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:06.398557 16361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:06.399196 16361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:06.400050 16361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:06.400098 16361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.400143 16361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:06.400166 16361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.408851 16361 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:36869
I20260619 02:35:06.408951 16489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:36869 every 8 connection(s)
I20260619 02:35:06.409992 16361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:35:06.413908 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 16361
I20260619 02:35:06.414042 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:35:06.414307 7947 external_mini_cluster.cc:1428] Setting key 4e15d1bf9767461c1d2e49b5a463c1e1
Jun 19 02:35:06 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836506, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:35:06.424345 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:06.411896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:36759 (local address 127.7.194.254:35519)
0619 02:35:06.412058 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:06.412069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:06.412927 (+ 858us) server_negotiation.cc:407] Connection header received
0619 02:35:06.414414 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:06.414417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:06.414475 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:06.414584 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:06.415951 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.416489 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.417682 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.417914 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.418023 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:06.420652 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:06.420655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:06.420657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:06.420688 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:06.422216 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:06.422737 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:06.422743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:06.422745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:06.422828 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:06.423172 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:06.423176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:06.423177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:06.423415 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:06.423520 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:06.424035 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 02:35:06.424176 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:35:06.425338 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:06.412301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:36759)
0619 02:35:06.412758 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:06.412803 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:06.414199 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:06.414759 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:06.414767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:06.415231 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:06.415819 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:06.415834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.416614 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.416618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:06.417521 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:06.417529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.418061 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.418066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:06.418256 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:06.418926 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:06.418944 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:06.420461 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:06.422334 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:06.422339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:06.422357 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:06.422630 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:06.422917 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:06.422921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:06.422923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:06.423070 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:06.423516 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:06.423523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:06.423799 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 02:35:06.423996 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 02:35:06.424412 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:35:06.426716 16490 heartbeater.cc:347] Connected to a master server at 127.7.194.254:35519
I20260619 02:35:06.426978 16490 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:06.427520 16490 heartbeater.cc:511] Master 127.7.194.254:35519 requested a full tablet report, sending...
I20260619 02:35:06.428637 16165 ts_manager.cc:194] Registered new tserver with Master: 4550c9003dac45cd8e7087b36722565b (127.7.194.194:36869)
I20260619 02:35:06.429284 16165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:36759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:35:06.442881 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:35519
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:06.561106 16495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:06.561425 16495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:06.561473 16495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:06.565833 16495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:06.565914 16495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:06.565977 16495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:35:06.571621 16495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:06.572772 16495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:06.573891 16495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:06.584425 16500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:06.584460 16503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:06.584463 16501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:06.585060 16495 server_base.cc:1034] running on GCE node
I20260619 02:35:06.585635 16495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:06.586530 16495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:06.587726 16495 hybrid_clock.cc:648] HybridClock initialized: now 1781836506587707 us; error 41 us; skew 500 ppm
Jun 19 02:35:06 dist-test-slave-9tkz krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836506, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:06.590962 16495 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:35:06.591948 16495 webserver.cc:460] Webserver started at http://127.7.194.195:40367/ using document root <none> and password file <none>
I20260619 02:35:06.592638 16495 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:06.592725 16495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:06.592958 16495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:06.595147 16495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "8e899de2df1b42a7b6dac253a1f55420"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "9d58a3e0d602809395172e4842bed6bb"
server_key_iv: "b57c6efd55b360f053c92b86e1c7bb49"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.595726 16495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "8e899de2df1b42a7b6dac253a1f55420"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "9d58a3e0d602809395172e4842bed6bb"
server_key_iv: "b57c6efd55b360f053c92b86e1c7bb49"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.599900 16495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:35:06.602608 16510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.603943 16495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:06.604100 16495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "8e899de2df1b42a7b6dac253a1f55420"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "9d58a3e0d602809395172e4842bed6bb"
server_key_iv: "b57c6efd55b360f053c92b86e1c7bb49"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.604226 16495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:06.630829 16495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:06.634545 16495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:06.634768 16495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:06.635407 16495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:06.636300 16495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:06.636372 16495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.636442 16495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:06.636484 16495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.646044 16495 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:44905
I20260619 02:35:06.646073 16623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:44905 every 8 connection(s)
I20260619 02:35:06.647037 16495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:35:06.650386 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 16495
I20260619 02:35:06.650535 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:35:06.650874 7947 external_mini_cluster.cc:1428] Setting key b77289cafc28aab9bf3d04626894fc91
Jun 19 02:35:06 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836506, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:35:06.662236 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:06.649137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:44469 (local address 127.7.194.254:35519)
0619 02:35:06.649309 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:06.649312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:06.650489 (+ 1177us) server_negotiation.cc:407] Connection header received
0619 02:35:06.652137 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:06.652140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:06.652216 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:06.652294 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:06.653700 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.654284 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.655453 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.655678 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.655758 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:06.658383 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:06.658386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:06.658388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:06.658412 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:06.659809 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:06.660453 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:06.660456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:06.660458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:06.660533 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:06.660941 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:06.660945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:06.660947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:06.661191 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:06.661278 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:06.661956 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 02:35:06.662105 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 02:35:06.663156 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:06.649685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:44469)
0619 02:35:06.650294 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:06.650362 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:06.651877 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:06.652498 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:06.652506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:06.652901 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:06.653510 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:06.653524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.654404 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.654407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:06.655341 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:06.655350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:06.655831 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:06.655835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:06.655984 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:06.656629 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:06.656646 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:06.658208 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:06.659943 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:06.659949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:06.659963 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:06.660333 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:06.660613 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:06.660616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:06.660618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:06.660808 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:06.661324 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:06.661330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:06.661687 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 02:35:06.661861 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 02:35:06.662193 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":136,"threads_started":1}
I20260619 02:35:06.664434 16624 heartbeater.cc:347] Connected to a master server at 127.7.194.254:35519
I20260619 02:35:06.664709 16624 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:06.665316 16624 heartbeater.cc:511] Master 127.7.194.254:35519 requested a full tablet report, sending...
I20260619 02:35:06.666391 16164 ts_manager.cc:194] Registered new tserver with Master: 8e899de2df1b42a7b6dac253a1f55420 (127.7.194.195:44905)
I20260619 02:35:06.667011 16164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:44469
I20260619 02:35:06.676954 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab.
Entry for principal oryx/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab.
WARNING: no policy specified for oryx/127.7.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab.
Entry for principal oryx/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab.
WARNING: no policy specified for oryx/127.7.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab.
Entry for principal oryx/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab.
I20260619 02:35:06.755705 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 16227
I20260619 02:35:06.761673 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 16361
I20260619 02:35:06.767189 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 16495
I20260619 02:35:06.772482 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 16133
I20260619 02:35:06.779053 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:35519
--webserver_interface=127.7.194.254
--webserver_port=36663
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:06.895670 16633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:06.896036 16633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:06.896123 16633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:06.900396 16633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:35:06.900512 16633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:06.900549 16633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:06.900593 16633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:35:06.900629 16633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:35:06.906270 16633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=36663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:06.907495 16633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:06.908674 16633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:06.917927 16639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:06.917927 16641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:06.918166 16638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:06.918443 16633 server_base.cc:1034] running on GCE node
I20260619 02:35:06.919111 16633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:06.920341 16633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:06.921586 16633 hybrid_clock.cc:648] HybridClock initialized: now 1781836506921517 us; error 47 us; skew 500 ppm
Jun 19 02:35:06 dist-test-slave-9tkz krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836506, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:06.924484 16633 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:35:06.925486 16633 webserver.cc:460] Webserver started at http://127.7.194.254:36663/ using document root <none> and password file <none>
I20260619 02:35:06.926118 16633 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:06.926229 16633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:06.930648 16633 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:35:06.933184 16648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:06.934580 16633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:06.934744 16633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "a690804f9cb0434ea1cf75d8ef2d080e"
format_stamp: "Formatted at 2026-06-19 02:35:05 on dist-test-slave-9tkz"
server_key: "f83b230b1dd409f0081e442f18c518cd"
server_key_iv: "aca615654b1ab9200a7f3b9fbb6de4bd"
server_key_version: "encryptionkey@0"
I20260619 02:35:06.935500 16633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:06.947693 16633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:06.951081 16633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:06.951294 16633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:06.958721 16633 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:35519
I20260619 02:35:06.958745 16700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:35519 every 8 connection(s)
I20260619 02:35:06.959890 16633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:35:06.964402 16701 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:35:06.965847 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 16633
I20260619 02:35:06.966310 16701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: Bootstrap starting.
I20260619 02:35:06.966477 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:40877
--local_ip_for_outbound_sockets=127.7.194.193
--tserver_master_addrs=127.7.194.254:35519
--webserver_port=45843
--webserver_interface=127.7.194.193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.194.193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:35:06.983306 16701 log.cc:824] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:06.990175 16701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:35:06.990602 16701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: Bootstrap complete.
I20260619 02:35:06.995031 16701 raft_consensus.cc:348] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } }
I20260619 02:35:06.995471 16701 raft_consensus.cc:729] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a690804f9cb0434ea1cf75d8ef2d080e, State: Initialized, Role: FOLLOWER
I20260619 02:35:06.996018 16701 consensus_queue.cc:260] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } }
I20260619 02:35:06.996158 16701 raft_consensus.cc:388] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:35:06.996244 16701 raft_consensus.cc:482] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:35:06.996354 16701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:35:06.998457 16701 raft_consensus.cc:504] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } }
I20260619 02:35:06.998800 16701 leader_election.cc:302] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a690804f9cb0434ea1cf75d8ef2d080e; no voters:
I20260619 02:35:06.999152 16701 leader_election.cc:288] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:35:06.999251 16705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:35:06.999442 16705 raft_consensus.cc:686] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [term 2 LEADER]: Becoming Leader. State: Replica: a690804f9cb0434ea1cf75d8ef2d080e, State: Running, Role: LEADER
I20260619 02:35:06.999747 16705 consensus_queue.cc:237] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } }
I20260619 02:35:07.000439 16701 sys_catalog.cc:564] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:35:07.001106 16706 sys_catalog.cc:455] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } } }
I20260619 02:35:07.001231 16706 sys_catalog.cc:458] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:07.001726 16712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:35:07.002434 16707 sys_catalog.cc:455] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a690804f9cb0434ea1cf75d8ef2d080e. Latest consensus state: current_term: 2 leader_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a690804f9cb0434ea1cf75d8ef2d080e" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35519 } } }
I20260619 02:35:07.002622 16707 sys_catalog.cc:458] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:07.005091 16712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:35:07.005759 16712 catalog_manager.cc:1259] Loaded cluster ID: 0f93abbd0b514812948a6f0ba497b159
I20260619 02:35:07.005820 16712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:35:07.008200 16712 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:35:07.008867 16712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a690804f9cb0434ea1cf75d8ef2d080e: Loaded TSK: 0
I20260619 02:35:07.010138 16712 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:35:07.085369 16703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:07.085723 16703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:07.085807 16703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:07.090004 16703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:07.090106 16703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:07.090201 16703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:35:07.095834 16703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.193.keytab
--principal=oryx/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:40877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=45843
--tserver_master_addrs=127.7.194.254:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:07.096959 16703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:07.098052 16703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:07.108453 16728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:07.108448 16727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:07.108616 16703 server_base.cc:1034] running on GCE node
W20260619 02:35:07.108950 16730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:07.109447 16703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:07.110208 16703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:07.111399 16703 hybrid_clock.cc:648] HybridClock initialized: now 1781836507111368 us; error 57 us; skew 500 ppm
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836507, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:07.114436 16703 init.cc:377] Logged in from keytab as oryx/127.7.194.193@KRBTEST.COM (short username oryx)
I20260619 02:35:07.115489 16703 webserver.cc:460] Webserver started at http://127.7.194.193:45843/ using document root <none> and password file <none>
I20260619 02:35:07.116087 16703 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:07.116163 16703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:07.120476 16703 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:35:07.122856 16737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:07.124244 16703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:07.124382 16703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "916f748c74af459da77ba6d121fb210e"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "d654462f7d748b74fa6ca8b6afe78cb6"
server_key_iv: "a7daca2a43f3ad9e66f85ae630934818"
server_key_version: "encryptionkey@0"
I20260619 02:35:07.124954 16703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:07.149206 16703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:07.152417 16703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:07.152734 16703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:07.153440 16703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:07.154330 16703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:07.154378 16703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:07.154455 16703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:07.154485 16703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:07.163823 16703 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:40877
I20260619 02:35:07.163841 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:40877 every 8 connection(s)
I20260619 02:35:07.164921 16703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:35:07.173763 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 16703
I20260619 02:35:07.174331 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:36869
--local_ip_for_outbound_sockets=127.7.194.194
--tserver_master_addrs=127.7.194.254:35519
--webserver_port=36515
--webserver_interface=127.7.194.194
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.194.194
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:35:07.178784 16853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:07.167085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:44967 (local address 127.7.194.254:35519)
0619 02:35:07.167561 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.167590 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.167898 (+ 308us) server_negotiation.cc:407] Connection header received
0619 02:35:07.169330 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.169341 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.169704 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.169982 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.171073 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.172124 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.173237 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.173970 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.174149 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.177960 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:44967: BlockingRecv error: recv got EOF from 127.7.194.193:44967 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":101,"threads_started":1}
I20260619 02:35:07.178853 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.167261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:44967)
0619 02:35:07.167741 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:07.167784 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:07.168949 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:07.170075 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:07.170082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:07.170462 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:07.170923 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.170937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.172242 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.172244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.173113 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.173121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.174137 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.174140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.174294 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.175007 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:07.175025 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:07.177749 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
W20260619 02:35:07.179566 16851 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:35519 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.194.254:35519: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:07.185374 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.180061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:51317)
0619 02:35:07.180185 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:07.180200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:07.180289 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:07.180522 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:07.180525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:07.180697 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:07.180792 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.180797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.181526 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.181533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.182113 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.182123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.182564 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.182576 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.182671 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.183208 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:07.183220 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:07.185222 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:35:07.185475 16853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:07.180162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:51317 (local address 127.7.194.254:35519)
0619 02:35:07.180293 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.180297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.180313 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:35:07.180374 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.180377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.180426 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.180506 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.180902 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.181423 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.182230 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.182458 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.182564 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.185352 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:51317: BlockingRecv error: recv got EOF from 127.7.194.193:51317 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:07.191538 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.186148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:56425)
0619 02:35:07.186259 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:07.186274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:07.186350 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:07.186610 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:07.186612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:07.186737 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:07.186811 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.186815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.187812 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.187814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.188331 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.188337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.188877 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.188880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.188941 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.189432 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:07.189444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:07.191357 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 02:35:07.191570 16853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:07.186209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:56425 (local address 127.7.194.254:35519)
0619 02:35:07.186322 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.186326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.186341 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:35:07.186435 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.186438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.186485 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.186563 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.186903 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.187714 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.188426 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.188775 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.188877 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.191477 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:56425: BlockingRecv error: recv got EOF from 127.7.194.193:56425 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 02:35:07.191867 16851 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:35:07.292330 16855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:07.292646 16855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:07.292694 16855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:07.296964 16855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:07.297050 16855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:07.297122 16855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:35:07.303077 16855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.194.keytab
--principal=oryx/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:36869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=36515
--tserver_master_addrs=127.7.194.254:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:07.304270 16855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:07.305404 16855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:07.315897 16860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:07.315961 16863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:07.315949 16861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:07.316295 16855 server_base.cc:1034] running on GCE node
I20260619 02:35:07.316843 16855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:07.317507 16855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:07.318714 16855 hybrid_clock.cc:648] HybridClock initialized: now 1781836507318693 us; error 34 us; skew 500 ppm
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836507, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:07.321713 16855 init.cc:377] Logged in from keytab as oryx/127.7.194.194@KRBTEST.COM (short username oryx)
I20260619 02:35:07.322681 16855 webserver.cc:460] Webserver started at http://127.7.194.194:36515/ using document root <none> and password file <none>
I20260619 02:35:07.323273 16855 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:07.323366 16855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:07.327426 16855 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:35:07.329802 16870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:07.331099 16855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:35:07.331238 16855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "4550c9003dac45cd8e7087b36722565b"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "643ffb95bd4d6c363704639f8e49ebcb"
server_key_iv: "935d7a1f91996a0c8168d34743ea219d"
server_key_version: "encryptionkey@0"
I20260619 02:35:07.331801 16855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:07.357372 16855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:07.361042 16855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:07.361266 16855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:07.362025 16855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:07.363139 16855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:07.363209 16855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:07.363279 16855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:07.363312 16855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:07.372607 16855 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:36869
I20260619 02:35:07.372632 16983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:36869 every 8 connection(s)
I20260619 02:35:07.373752 16855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:35:07.382268 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 16855
I20260619 02:35:07.382759 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:44905
--local_ip_for_outbound_sockets=127.7.194.195
--tserver_master_addrs=127.7.194.254:35519
--webserver_port=40367
--webserver_interface=127.7.194.195
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.194.195
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:35:07.385452 16853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:07.375766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:38033 (local address 127.7.194.254:35519)
0619 02:35:07.375903 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.375907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.376824 (+ 917us) server_negotiation.cc:407] Connection header received
0619 02:35:07.378153 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.378156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.378211 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.378292 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.379521 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.380094 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.381214 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.381450 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.381608 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.385322 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:38033: BlockingRecv error: recv got EOF from 127.7.194.194:38033 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:35:07.386309 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.376216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:38033)
0619 02:35:07.376658 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:07.376703 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:07.377925 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:07.378466 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:07.378473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:07.378847 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:07.379357 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.379371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.380202 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.380204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.381090 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.381097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.381628 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.381632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.381794 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.382482 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:07.382500 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:07.385194 (+ 2694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
W20260619 02:35:07.386839 16984 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:35519 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.194.254:35519: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:07.392974 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.387309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:40689)
0619 02:35:07.387413 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:07.387427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:07.387509 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:07.387752 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:07.387755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:07.387919 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:07.388013 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.388018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.388762 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.388768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.389315 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.389323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.390065 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.390075 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.390195 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.390723 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:07.390735 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:07.392763 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 02:35:07.393015 16853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:07.387405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:40689 (local address 127.7.194.254:35519)
0619 02:35:07.387531 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.387537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.387554 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:35:07.387599 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.387601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.387648 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.387732 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.388130 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.388645 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.389429 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.389850 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.390061 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.392918 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:40689: BlockingRecv error: recv got EOF from 127.7.194.194:40689 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:07.399719 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.393895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:50359)
0619 02:35:07.394116 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:07.394128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:07.394207 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:07.394474 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:07.394476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:07.394626 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:07.394705 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.394710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.395754 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.395757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.396339 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.396349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.396937 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.396940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.397007 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.397630 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:07.397643 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:07.399607 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158}
W20260619 02:35:07.400022 16984 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:35:07.400032 16853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:07.393902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:50359 (local address 127.7.194.254:35519)
0619 02:35:07.394008 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.394013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.394199 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:35:07.394283 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.394289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.394350 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.394453 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.394806 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.395637 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.396454 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.396797 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.396897 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.399774 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:50359: BlockingRecv error: recv got EOF from 127.7.194.194:50359 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 02:35:07.503265 16987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:07.503571 16987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:07.503654 16987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:07.508188 16987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:07.508268 16987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:07.508373 16987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:35:07.514593 16987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/krb5kdc/oryx_127.7.194.195.keytab
--principal=oryx/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:44905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=40367
--tserver_master_addrs=127.7.194.254:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:07.515762 16987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:07.516930 16987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:07.526981 16992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:07.526981 16995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:07.527078 16993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:07.527220 16987 server_base.cc:1034] running on GCE node
I20260619 02:35:07.527689 16987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:07.528414 16987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:07.529657 16987 hybrid_clock.cc:648] HybridClock initialized: now 1781836507529584 us; error 64 us; skew 500 ppm
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836507, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:07.532707 16987 init.cc:377] Logged in from keytab as oryx/127.7.194.195@KRBTEST.COM (short username oryx)
I20260619 02:35:07.533715 16987 webserver.cc:460] Webserver started at http://127.7.194.195:40367/ using document root <none> and password file <none>
I20260619 02:35:07.534273 16987 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:07.534346 16987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:07.538746 16987 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 02:35:07.541071 17002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:07.542527 16987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:07.542686 16987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "8e899de2df1b42a7b6dac253a1f55420"
format_stamp: "Formatted at 2026-06-19 02:35:06 on dist-test-slave-9tkz"
server_key: "9d58a3e0d602809395172e4842bed6bb"
server_key_iv: "b57c6efd55b360f053c92b86e1c7bb49"
server_key_version: "encryptionkey@0"
I20260619 02:35:07.543280 16987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:07.558652 16987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:07.561923 16987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:07.562132 16987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:07.562763 16987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:07.563648 16987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:07.563716 16987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:07.563796 16987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:07.563835 16987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:07.573184 16987 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:44905
I20260619 02:35:07.573203 17115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:44905 every 8 connection(s)
I20260619 02:35:07.574386 16987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:35:07.580827 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 16987
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:35:07.586795 16853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:07.576341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:45277 (local address 127.7.194.254:35519)
0619 02:35:07.576483 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.576486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.577792 (+ 1306us) server_negotiation.cc:407] Connection header received
0619 02:35:07.578795 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.578798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.578859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.578963 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.580257 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.580803 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.582766 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.583017 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.583126 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.586666 (+ 3540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:45277: BlockingRecv error: recv got EOF from 127.7.194.195:45277 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:35:07.587558 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.576790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:45277)
0619 02:35:07.577519 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:07.577664 (+ 145us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:07.578571 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:07.579174 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:07.579182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:07.579584 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:07.580107 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.580121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.580952 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.580956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.582614 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.582626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.583183 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.583187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.583370 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.583937 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:07.583954 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:07.586558 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":108,"threads_started":1}
W20260619 02:35:07.588146 17116 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:35519 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.194.254:35519: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
I20260619 02:35:07.592060 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.582390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.7.194.254:35519)
0619 02:35:07.582638 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.582641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.582657 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:35:07.582692 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.582694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.582746 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.582863 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.583946 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.584501 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.585419 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.585716 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.585821 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.587345 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:07.587351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:07.587369 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:07.587398 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:07.589703 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:07.590161 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:07.590167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:07.590174 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:07.590336 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:07.590639 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:07.590641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:07.590644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:07.591136 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:07.591330 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:07.591580 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 02:35:07.591867 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:07.594789 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.588949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:54615)
0619 02:35:07.589099 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:07.589115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:07.589199 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:07.589434 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:07.589437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:07.589651 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:07.589747 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.589770 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.590690 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.590697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.591488 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.591499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.591981 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.591984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.592084 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.592564 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:07.592581 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:07.594636 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 02:35:07.594867 16853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:07.589059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:54615 (local address 127.7.194.254:35519)
0619 02:35:07.589179 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.589183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.589195 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:35:07.589292 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.589295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.589337 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.589417 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.589882 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.590560 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.591620 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.591875 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.591955 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.594749 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:54615: BlockingRecv error: recv got EOF from 127.7.194.195:54615 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:07 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:07.600514 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:07.595594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:51731)
0619 02:35:07.595690 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:07.595721 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:07.595800 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:07.596068 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:07.596071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:07.596236 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:07.596302 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.596307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.597021 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.597024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.597643 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:07.597652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.598086 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.598090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:07.598167 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.598588 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:07.598600 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:07.600359 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 02:35:07.600595 16853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:07.595655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:51731 (local address 127.7.194.254:35519)
0619 02:35:07.595799 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:07.595803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:07.595816 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:35:07.595890 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:07.595893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:07.595931 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:07.596055 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:07.596406 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.596907 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.597750 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:07.597984 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:07.598087 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:07.600468 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:51731: BlockingRecv error: recv got EOF from 127.7.194.195:51731 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 02:35:07.600876 17116 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 02:35:08 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:08 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:08.198987 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:08.192603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:42949)
0619 02:35:08.192905 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:08.192922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:08.193033 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:08.193306 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:08.193308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:08.193512 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:08.193642 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:08.193648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.194619 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.194627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:08.195641 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:08.195655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.196136 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.196138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:08.196206 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:08.196817 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:08.196839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:08.198839 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":115,"threads_started":1}
W20260619 02:35:08.199150 17122 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:08.192737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:42949 (local address 127.7.194.254:35519)
0619 02:35:08.192992 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:08.192996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:08.193014 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:35:08.193123 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:08.193126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:08.193170 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:08.193260 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:08.193798 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.194469 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.195780 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.196004 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.196098 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:08.198994 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:42949: BlockingRecv error: recv got EOF from 127.7.194.193:42949 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":113,"threads_started":1}
Jun 19 02:35:08 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:08 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:08.406353 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:08.400643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:51863)
0619 02:35:08.400965 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:08.400979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:08.401108 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:08.401392 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:08.401394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:08.401623 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:08.401779 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:08.401796 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.402615 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.402618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:08.403217 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:08.403225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.403662 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.403665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:08.403735 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:08.404282 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:08.404293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:08.406218 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":119,"threads_started":1}
W20260619 02:35:08.406527 17122 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:08.400693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:51863 (local address 127.7.194.254:35519)
0619 02:35:08.400823 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:08.400827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:08.401094 (+ 267us) server_negotiation.cc:407] Connection header received
0619 02:35:08.401207 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:08.401210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:08.401254 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:08.401322 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:08.401927 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.402478 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.403331 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.403552 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.403631 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:08.406391 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:51863: BlockingRecv error: recv got EOF from 127.7.194.194:51863 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 02:35:08 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:08 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:08.607977 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:08.601696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:35231)
0619 02:35:08.602091 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:08.602105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:08.602249 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:08.602539 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:08.602542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:08.602724 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:08.602822 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:08.602827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.603834 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.603837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:08.604645 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:08.604655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.605329 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.605332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:08.605396 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:08.605990 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:08.606002 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:08.607838 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
W20260619 02:35:08.608193 17122 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:08.601704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:35231 (local address 127.7.194.254:35519)
0619 02:35:08.601912 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:08.601917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:08.602211 (+ 294us) server_negotiation.cc:407] Connection header received
0619 02:35:08.602344 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:08.602348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:08.602403 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:08.602494 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:08.602943 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.603718 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.604757 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:08.605202 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:08.605305 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:08.608048 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:35231: BlockingRecv error: recv got EOF from 127.7.194.195:35231 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 02:35:09 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:09 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:09.206163 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:09.200113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:41319)
0619 02:35:09.200403 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:09.200419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:09.200563 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:09.200901 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:09.200904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:09.201198 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:09.201353 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:09.201364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.202300 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.202303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:09.203001 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:09.203011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.203425 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.203427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:09.203486 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:09.204066 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:09.204087 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:09.206016 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
W20260619 02:35:09.206249 17126 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:09.200154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:41319 (local address 127.7.194.254:35519)
0619 02:35:09.200438 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:09.200445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:09.200493 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:35:09.200682 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:09.200685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:09.200754 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:09.200855 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:09.201496 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.202197 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.203104 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.203327 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.203412 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:09.206142 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:41319: BlockingRecv error: recv got EOF from 127.7.194.193:41319 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
Jun 19 02:35:09 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:09 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:09.413220 17127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:09.407176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:44223)
0619 02:35:09.407521 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:09.407541 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:09.407700 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:09.408023 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:09.408026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:09.408275 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:09.408393 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:09.408401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.409242 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.409248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:09.409902 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:09.409913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.410406 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.410410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:09.410495 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:09.411099 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:09.411126 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:09.413069 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":88,"threads_started":1}
W20260619 02:35:09.413458 17126 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:09.407315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:44223 (local address 127.7.194.254:35519)
0619 02:35:09.407511 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:09.407517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:09.407671 (+ 154us) server_negotiation.cc:407] Connection header received
0619 02:35:09.407834 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:09.407837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:09.407903 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:09.408001 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:09.408528 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.409097 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.410033 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.410281 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.410405 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:09.413314 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:44223: BlockingRecv error: recv got EOF from 127.7.194.194:44223 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 02:35:09 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:09 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:09.614380 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:09.608791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:47311)
0619 02:35:09.609119 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:09.609133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:09.609231 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:09.609454 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:09.609456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:09.609675 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:09.609804 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:09.609810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.610547 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.610551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:09.611138 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:09.611146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.611569 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.611572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:09.611634 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:09.612161 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:09.612173 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:09.614149 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
W20260619 02:35:09.614399 17126 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:09.609009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:47311 (local address 127.7.194.254:35519)
0619 02:35:09.609126 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:09.609130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:09.609181 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:35:09.609301 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:09.609304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:09.609348 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:09.609416 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:09.609906 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.610448 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.611245 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:09.611472 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:09.611552 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:09.614291 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:47311: BlockingRecv error: recv got EOF from 127.7.194.195:47311 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 02:35:10 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:10 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:10.212667 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:10.207060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:39023)
0619 02:35:10.207361 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:10.207375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:10.207481 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:10.207778 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:10.207780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:10.207970 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:10.208072 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:10.208078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.208870 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.208873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:10.209448 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:10.209457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.209912 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.209915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:10.209979 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:10.210498 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:10.210522 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:10.212541 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
W20260619 02:35:10.212870 17130 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:10.207148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:39023 (local address 127.7.194.254:35519)
0619 02:35:10.207420 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:10.207424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:10.207439 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:35:10.207606 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:10.207609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:10.207655 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:10.207760 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:10.208176 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.208744 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.209581 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.209811 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.209893 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:10.212688 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:39023: BlockingRecv error: recv got EOF from 127.7.194.193:39023 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
Jun 19 02:35:10 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:10 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:10.419777 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:10.414103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:48429)
0619 02:35:10.414381 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:10.414396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:10.414496 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:10.414749 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:10.414751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:10.414930 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:10.415029 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:10.415034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.415921 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.415925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:10.416534 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:10.416543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.416971 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.416973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:10.417042 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:10.417655 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:10.417667 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:10.419627 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":116,"threads_started":1}
W20260619 02:35:10.419899 17130 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:10.414168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:48429 (local address 127.7.194.254:35519)
0619 02:35:10.414326 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:10.414331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:10.414471 (+ 140us) server_negotiation.cc:407] Connection header received
0619 02:35:10.414571 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:10.414574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:10.414622 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:10.414692 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:10.415186 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.415768 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.416652 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.416868 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.416950 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:10.419777 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:48429: BlockingRecv error: recv got EOF from 127.7.194.194:48429 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 02:35:10 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:10 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:10.620813 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:10.615169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:35933)
0619 02:35:10.615520 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:10.615545 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:10.615683 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:10.615957 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:10.615959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:10.616186 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:10.616281 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:10.616287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.617068 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.617073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:10.617730 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:10.617738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.618211 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.618215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:10.618304 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:10.618838 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:10.618850 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:10.620690 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":82,"threads_started":1}
W20260619 02:35:10.620927 17130 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:10.615293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:35933 (local address 127.7.194.254:35519)
0619 02:35:10.615473 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:10.615478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:10.615624 (+ 146us) server_negotiation.cc:407] Connection header received
0619 02:35:10.615785 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:10.615788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:10.615840 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:10.615935 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:10.616402 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.616947 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.617846 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:10.618097 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:10.618208 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:10.620807 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:35933: BlockingRecv error: recv got EOF from 127.7.194.195:35933 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 02:35:11 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:11 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:11.218787 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:11.213471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:48885)
0619 02:35:11.213837 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:11.213855 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:11.213957 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:11.214186 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:11.214188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:11.214360 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:11.214457 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:11.214462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.215193 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.215195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:11.215749 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:11.215757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.216192 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.216194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:11.216257 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:11.216765 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:11.216777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:11.218665 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":164,"threads_started":1}
W20260619 02:35:11.219015 17134 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:11.213694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:48885 (local address 127.7.194.254:35519)
0619 02:35:11.213945 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:11.213949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:11.213962 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:35:11.214026 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:11.214029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:11.214074 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:11.214151 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:11.214584 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.215102 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.215858 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.216097 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.216176 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:11.218838 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:48885: BlockingRecv error: recv got EOF from 127.7.194.193:48885 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
Jun 19 02:35:11 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:11 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:11.433938 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:11.420693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:33895)
0619 02:35:11.420998 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:11.421011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:11.421145 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:11.421411 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:11.421414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:11.421654 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:11.421752 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:11.421757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.422539 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.422545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:11.423130 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:11.423137 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.423595 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.423599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:11.423688 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:11.431764 (+ 8076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:11.431779 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:11.433810 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
W20260619 02:35:11.434146 17134 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:11.420762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:33895 (local address 127.7.194.254:35519)
0619 02:35:11.420929 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:11.420935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:11.421088 (+ 153us) server_negotiation.cc:407] Connection header received
0619 02:35:11.421242 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:11.421246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:11.421299 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:11.421387 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:11.421872 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.422424 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.423247 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.423476 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.423588 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:11.434008 (+ 10420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:33895: BlockingRecv error: recv got EOF from 127.7.194.194:33895 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 02:35:11 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:11 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:11.627190 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:11.621729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:54781)
0619 02:35:11.621995 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:11.622014 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:11.622110 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:11.622354 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:11.622356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:11.622565 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:11.622663 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:11.622669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.623457 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.623460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:11.624030 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:11.624039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.624516 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.624518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:11.624586 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:11.625113 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:11.625126 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:11.627058 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
W20260619 02:35:11.627331 17134 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:11.621788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:54781 (local address 127.7.194.254:35519)
0619 02:35:11.621931 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:11.621938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:11.622067 (+ 129us) server_negotiation.cc:407] Connection header received
0619 02:35:11.622194 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:11.622197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:11.622243 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:11.622315 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:11.622774 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.623314 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.624163 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:11.624396 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:11.624485 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:11.627209 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:54781: BlockingRecv error: recv got EOF from 127.7.194.195:54781 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 02:35:12 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:12 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:12.224972 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:12.219707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:37627)
0619 02:35:12.219996 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:12.220010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:12.220092 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:12.220329 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:12.220331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:12.220533 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:12.220623 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:12.220628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.221414 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.221417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:12.222053 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:12.222062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.222477 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.222480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:12.222540 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:12.223034 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:12.223045 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:12.224849 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
W20260619 02:35:12.225162 17138 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:12.219759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:37627 (local address 127.7.194.254:35519)
0619 02:35:12.220048 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:12.220051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:12.220064 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:35:12.220173 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:12.220175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:12.220218 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:12.220295 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:12.220753 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.221321 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.222150 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.222371 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.222453 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:12.225010 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:37627: BlockingRecv error: recv got EOF from 127.7.194.193:37627 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
Jun 19 02:35:12 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:12 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:12.440794 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:12.434862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:41329)
0619 02:35:12.435160 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:12.435176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:12.435302 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:12.435658 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:12.435660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:12.435846 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:12.435955 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:12.435961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.436862 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.436864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:12.437430 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:12.437438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.437897 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.437899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:12.437971 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:12.438512 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:12.438524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:12.440671 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
W20260619 02:35:12.441037 17138 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:12.434992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:41329 (local address 127.7.194.254:35519)
0619 02:35:12.435146 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:12.435151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:12.435248 (+ 97us) server_negotiation.cc:407] Connection header received
0619 02:35:12.435473 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:12.435476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:12.435525 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:12.435626 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:12.436104 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.436762 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.437568 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.437800 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.437882 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:12.440911 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:41329: BlockingRecv error: recv got EOF from 127.7.194.194:41329 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 02:35:12 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:12 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:12.634084 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:12.628040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:39617)
0619 02:35:12.628383 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:12.628396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:12.628515 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:12.628748 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:12.628751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:12.628999 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:12.629096 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:12.629102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.630097 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.630100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:12.630686 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:12.630695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.631175 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.631177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:12.631239 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:12.631803 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:12.631815 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:12.633958 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
W20260619 02:35:12.634310 17138 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:12.628133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:39617 (local address 127.7.194.254:35519)
0619 02:35:12.628266 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:12.628271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:12.628488 (+ 217us) server_negotiation.cc:407] Connection header received
0619 02:35:12.628601 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:12.628603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:12.628653 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:12.628715 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:12.629313 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.629985 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.630837 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:12.631064 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:12.631159 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:12.634112 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:39617: BlockingRecv error: recv got EOF from 127.7.194.195:39617 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 02:35:13 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:13 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:13.232035 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:13.226016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:53847)
0619 02:35:13.226404 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:13.226418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:13.226531 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:13.226881 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:13.226884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:13.227086 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:13.227189 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:13.227194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.228062 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.228065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:13.228673 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:13.228682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.229217 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.229220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:13.229297 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:13.229937 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:13.229951 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:13.231906 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":168,"threads_started":1}
W20260619 02:35:13.232461 17141 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:13.226093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:53847 (local address 127.7.194.254:35519)
0619 02:35:13.226413 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:13.226417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:13.226500 (+ 83us) server_negotiation.cc:407] Connection header received
0619 02:35:13.226665 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:13.226668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:13.226722 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:13.226818 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:13.227327 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.227919 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.228844 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.229093 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.229188 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:13.232309 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:53847: BlockingRecv error: recv got EOF from 127.7.194.193:53847 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
Jun 19 02:35:13 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:13 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:13.447273 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:13.441622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:52883)
0619 02:35:13.441912 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:13.441926 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:13.442035 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:13.442296 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:13.442299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:13.442489 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:13.442590 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:13.442594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.443337 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.443340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:13.443928 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:13.443936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.444382 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.444389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:13.444459 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:13.445071 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:13.445084 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:13.447093 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":107,"threads_started":1}
W20260619 02:35:13.447365 17141 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:13.441641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:52883 (local address 127.7.194.254:35519)
0619 02:35:13.441785 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:13.441789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:13.441995 (+ 206us) server_negotiation.cc:407] Connection header received
0619 02:35:13.442104 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:13.442107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:13.442151 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:13.442222 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:13.442696 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.443221 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.444036 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.444254 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.444337 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:13.447233 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:52883: BlockingRecv error: recv got EOF from 127.7.194.194:52883 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 02:35:13 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:13 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:13.640852 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:13.634906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:39747)
0619 02:35:13.635178 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:13.635192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:13.635309 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:13.635598 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:13.635600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:13.635788 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:13.635886 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:13.635891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.636750 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.636754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:13.637336 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:13.637345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.637840 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.637843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:13.637917 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:13.638551 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:13.638564 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:13.640721 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
W20260619 02:35:13.640995 17141 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:13.634961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:39747 (local address 127.7.194.254:35519)
0619 02:35:13.635093 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:13.635096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:13.635259 (+ 163us) server_negotiation.cc:407] Connection header received
0619 02:35:13.635410 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:13.635412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:13.635462 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:13.635563 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:13.635987 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.636598 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.637443 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:13.637726 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:13.637810 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:13.640867 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:39747: BlockingRecv error: recv got EOF from 127.7.194.195:39747 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 02:35:14 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:14 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:14.240116 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:14.233172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:59123)
0619 02:35:14.233571 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:14.233589 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:14.233722 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:14.234022 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:14.234025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:14.234270 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:14.234370 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:14.234377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.235186 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.235192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:14.235783 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:14.235792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.236353 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.236361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:14.236525 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:14.237248 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:14.237264 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:14.239868 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":103,"threads_started":1}
W20260619 02:35:14.240142 17146 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:14.233282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:59123 (local address 127.7.194.254:35519)
0619 02:35:14.233606 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:14.233615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:14.233657 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:35:14.233837 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:14.233841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:14.233899 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:14.234000 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:14.234505 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.235068 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.235910 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.236195 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.236310 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:14.240022 (+ 3712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:59123: BlockingRecv error: recv got EOF from 127.7.194.193:59123 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
Jun 19 02:35:14 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:14 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:14.454424 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:14.448385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:33337)
0619 02:35:14.448748 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:14.448764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:14.448878 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:14.449124 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:14.449126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:14.449353 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:14.449475 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:14.449481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.450369 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.450373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:14.450954 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:14.450962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.451540 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.451544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:14.451623 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:14.452288 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:14.452300 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:14.454286 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":102,"threads_started":1}
W20260619 02:35:14.454635 17146 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:14.448458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:33337 (local address 127.7.194.254:35519)
0619 02:35:14.448626 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:14.448631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:14.448871 (+ 240us) server_negotiation.cc:407] Connection header received
0619 02:35:14.448952 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:14.448955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:14.449005 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:14.449086 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:14.449633 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.450244 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.451140 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.451378 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.451468 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:14.454493 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:33337: BlockingRecv error: recv got EOF from 127.7.194.194:33337 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 02:35:14 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:14 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:14.647480 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:14.641791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:43649)
0619 02:35:14.642096 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:14.642111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:14.642215 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:14.642460 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:14.642462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:14.642653 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:14.642749 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:14.642755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.643566 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.643570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:14.644187 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:14.644196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.644627 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.644629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:14.644697 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:14.645306 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:14.645319 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:14.647287 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":92,"threads_started":1}
W20260619 02:35:14.647560 17146 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:14.641907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:43649 (local address 127.7.194.254:35519)
0619 02:35:14.642046 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:14.642049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:14.642180 (+ 131us) server_negotiation.cc:407] Connection header received
0619 02:35:14.642295 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:14.642297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:14.642345 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:14.642424 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:14.642861 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.643448 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.644297 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:14.644515 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:14.644608 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:14.647443 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:43649: BlockingRecv error: recv got EOF from 127.7.194.195:43649 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 02:35:15 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:15 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:15.246372 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:15.240899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:47607)
0619 02:35:15.241173 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:15.241187 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:15.241278 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:15.241508 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:15.241510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:15.241744 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:15.241843 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:15.241849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.242608 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.242610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:15.243204 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:15.243212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.243755 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.243759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:15.243830 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:15.244353 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:15.244365 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:15.246252 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
W20260619 02:35:15.246575 17150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:15.241045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:47607 (local address 127.7.194.254:35519)
0619 02:35:15.241268 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:15.241271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:15.241283 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:35:15.241345 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:15.241348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:15.241390 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:15.241474 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:15.241950 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.242511 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.243321 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.243561 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.243644 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:15.246406 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:47607: BlockingRecv error: recv got EOF from 127.7.194.193:47607 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
Jun 19 02:35:15 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:15 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:35:15.461776 17150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:15.455525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:36767 (local address 127.7.194.254:35519)
0619 02:35:15.455691 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:15.455696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:15.455885 (+ 189us) server_negotiation.cc:407] Connection header received
0619 02:35:15.456068 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:15.456073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:15.456152 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:15.456233 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:15.456868 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.457716 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.458529 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.458768 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.458863 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:15.461672 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:36767: BlockingRecv error: recv got EOF from 127.7.194.194:36767 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:35:15.461791 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:15.455329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:36767)
0619 02:35:15.455783 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:15.455797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:15.455953 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:15.456274 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:15.456277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:15.456535 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:15.456675 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:15.456684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.457820 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.457822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:15.458409 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:15.458417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.458886 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.458889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:15.458959 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:15.459583 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:15.459595 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:15.461494 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":100,"threads_started":1}
Jun 19 02:35:15 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:15 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:15.654284 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:15.648404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:36139)
0619 02:35:15.648686 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:15.648700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:15.648803 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:15.649032 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:15.649034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:15.649216 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:15.649309 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:15.649314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.650053 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.650056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:15.650906 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:15.650916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.651603 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.651606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:15.651714 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:15.652206 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:15.652218 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:15.654161 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
W20260619 02:35:15.654551 17150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:15.648460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:36139 (local address 127.7.194.254:35519)
0619 02:35:15.648601 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:15.648606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:15.648799 (+ 193us) server_negotiation.cc:407] Connection header received
0619 02:35:15.648871 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:15.648873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:15.648926 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:15.649000 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:15.649415 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.649967 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.651039 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:15.651475 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:15.651596 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:15.654400 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:36139: BlockingRecv error: recv got EOF from 127.7.194.195:36139 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 02:35:16 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:16 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:16.253206 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:16.247161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:37157)
0619 02:35:16.247545 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:16.247559 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:16.247706 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:16.247951 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:16.247953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:16.248131 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:16.248226 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:16.248232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.249085 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.249088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:16.249867 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:16.249882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.250367 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.250370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:16.250444 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:16.251037 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:16.251049 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:16.253063 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":84,"threads_started":1}
W20260619 02:35:16.253392 17154 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:16.247296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:37157 (local address 127.7.194.254:35519)
0619 02:35:16.247539 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:16.247543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:16.247643 (+ 100us) server_negotiation.cc:407] Connection header received
0619 02:35:16.247780 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:16.247783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:16.247832 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:16.247918 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:16.248357 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.248943 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.250006 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.250240 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.250335 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:16.253212 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:37157: BlockingRecv error: recv got EOF from 127.7.194.193:37157 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
Jun 19 02:35:16 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:16 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:16.472050 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:16.462660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:38279)
0619 02:35:16.462989 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:16.463005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:16.463147 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:16.463462 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:16.463465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:16.463686 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:16.463788 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:16.463793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.464580 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.464585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:16.465174 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:16.465182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.465729 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.465732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:16.465835 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:16.469824 (+ 3989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:16.469837 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:16.471861 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
W20260619 02:35:16.472157 17154 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:16.462721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:38279 (local address 127.7.194.254:35519)
0619 02:35:16.462852 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:16.462857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:16.463093 (+ 236us) server_negotiation.cc:407] Connection header received
0619 02:35:16.463272 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:16.463292 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:16.463349 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:16.463436 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:16.463912 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.464465 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.465305 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.465600 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.465702 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:16.472031 (+ 6329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:38279: BlockingRecv error: recv got EOF from 127.7.194.194:38279 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 02:35:16 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:16 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:16.660746 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:16.655234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:33653)
0619 02:35:16.655585 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:16.655599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:16.655707 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:16.655913 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:16.655915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:16.656102 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:16.656197 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:16.656202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.656925 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.656928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:16.657489 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:16.657497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.658016 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.658019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:16.658091 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:16.658602 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:16.658614 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:16.660596 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
W20260619 02:35:16.660885 17154 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:16.655263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:33653 (local address 127.7.194.254:35519)
0619 02:35:16.655417 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:16.655423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:16.655673 (+ 250us) server_negotiation.cc:407] Connection header received
0619 02:35:16.655788 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:16.655791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:16.655840 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:16.655901 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:16.656290 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.656815 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.657660 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:16.657877 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:16.657980 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:16.660756 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:33653: BlockingRecv error: recv got EOF from 127.7.194.195:33653 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:35:17 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:17 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:17.260382 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:17.254209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:56049)
0619 02:35:17.254504 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:17.254520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:17.254636 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:17.254906 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:17.254909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:17.255098 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:17.255202 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:17.255208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.256126 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.256130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:17.256906 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:17.256918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.257482 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.257485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:17.257617 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:17.258166 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:17.258179 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:17.260255 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":109,"threads_started":1}
W20260619 02:35:17.260596 17158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:17.254293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:56049 (local address 127.7.194.254:35519)
0619 02:35:17.254550 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:17.254553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:17.254590 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:35:17.254728 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:17.254731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:17.254774 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:17.254886 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:17.255334 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.255994 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.257074 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.257335 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.257434 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:17.260408 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:56049: BlockingRecv error: recv got EOF from 127.7.194.193:56049 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
Jun 19 02:35:17 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:17 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:17.478964 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:17.473150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:54283)
0619 02:35:17.473520 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:17.473534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:17.473707 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:17.474170 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:17.474173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:17.474428 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:17.474527 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:17.474534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.475279 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.475281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:17.475861 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:17.475869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.476298 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.476301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:17.476374 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:17.476986 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:17.476998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:17.478813 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":107,"threads_started":1}
W20260619 02:35:17.479091 17158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:17.473213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:54283 (local address 127.7.194.254:35519)
0619 02:35:17.473389 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:17.473396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:17.473617 (+ 221us) server_negotiation.cc:407] Connection header received
0619 02:35:17.473848 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:17.473851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:17.473910 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:17.474003 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:17.474630 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.475168 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.475957 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.476174 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.476251 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:17.478966 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:54283: BlockingRecv error: recv got EOF from 127.7.194.194:54283 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 02:35:17 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:17 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:17.668251 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:17.661831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:54589)
0619 02:35:17.662221 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:17.662240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:17.662421 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:17.662763 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:17.662766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:17.662996 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:17.663098 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:17.663105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.663977 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.663982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:17.664623 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:17.664631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.665156 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.665160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:17.665259 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:17.665899 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:17.665912 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:17.668131 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":102,"threads_started":1}
W20260619 02:35:17.668511 17158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:17.662011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:54589 (local address 127.7.194.254:35519)
0619 02:35:17.662223 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:17.662230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:17.662346 (+ 116us) server_negotiation.cc:407] Connection header received
0619 02:35:17.662541 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:17.662545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:17.662610 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:17.662722 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:17.663240 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.663844 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.664761 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:17.665019 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:17.665151 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:17.668372 (+ 3221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:54589: BlockingRecv error: recv got EOF from 127.7.194.195:54589 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 19 02:35:18 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:18 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:18.266824 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:18.261237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:39379)
0619 02:35:18.261510 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:18.261523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:18.261675 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:18.261901 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:18.261903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:18.262085 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:18.262179 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:18.262185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.263026 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.263028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:18.263598 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:18.263606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.264064 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.264067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:18.264130 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:18.264737 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:18.264752 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:18.266699 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":99,"threads_started":1}
W20260619 02:35:18.267086 17162 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:18.261332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:39379 (local address 127.7.194.254:35519)
0619 02:35:18.261581 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:18.261584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:18.261656 (+ 72us) server_negotiation.cc:407] Connection header received
0619 02:35:18.261752 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:18.261754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:18.261798 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:18.261868 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:18.262347 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.262916 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.263722 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.263958 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.264047 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:18.266938 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:39379: BlockingRecv error: recv got EOF from 127.7.194.193:39379 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
Jun 19 02:35:18 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:18 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:18.486279 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:18.480021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:48063)
0619 02:35:18.480374 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:18.480390 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:18.480541 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:18.480875 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:18.480878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:18.481115 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:18.481212 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:18.481218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.482127 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.482131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:18.482733 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:18.482742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.483223 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.483226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:18.483314 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:18.483891 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:18.483904 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:18.486140 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
W20260619 02:35:18.486591 17162 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:18.480169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:48063 (local address 127.7.194.254:35519)
0619 02:35:18.480341 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:18.480347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:18.480481 (+ 134us) server_negotiation.cc:407] Connection header received
0619 02:35:18.480699 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:18.480702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:18.480760 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:18.480852 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:18.481346 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.481970 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.482859 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.483098 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.483214 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:18.486346 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:48063: BlockingRecv error: recv got EOF from 127.7.194.194:48063 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 02:35:18 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:18 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:18.674860 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:18.669135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:51709)
0619 02:35:18.669435 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:18.669458 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:18.669610 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:18.669960 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:18.669965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:18.670210 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:18.670352 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:18.670361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.671125 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.671128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:18.671731 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:18.671739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.672165 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.672168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:18.672230 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:18.672744 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:18.672756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:18.674736 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
W20260619 02:35:18.675055 17162 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:18.669189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:51709 (local address 127.7.194.254:35519)
0619 02:35:18.669342 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:18.669348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:18.669577 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:35:18.669752 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:18.669756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:18.669826 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:18.669909 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:18.670484 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.671016 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.671842 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:18.672061 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:18.672159 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:18.674901 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:51709: BlockingRecv error: recv got EOF from 127.7.194.195:51709 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 02:35:19 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:19 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:19.273648 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:19.268044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:43091)
0619 02:35:19.268352 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:19.268366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:19.268471 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:19.268726 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:19.268728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:19.268900 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:19.268995 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:19.269000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.269772 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.269775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:19.270323 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:19.270330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.270893 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.270897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:19.270987 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:19.271579 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:19.271591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:19.273443 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":76,"threads_started":1}
W20260619 02:35:19.273730 17166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:19.268110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:43091 (local address 127.7.194.254:35519)
0619 02:35:19.268419 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:19.268423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:19.268438 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:35:19.268559 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:19.268562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:19.268606 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:19.268691 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:19.269100 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.269665 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.270457 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.270775 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.270888 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:19.273601 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:43091: BlockingRecv error: recv got EOF from 127.7.194.193:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
Jun 19 02:35:19 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:19 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:19.492898 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:19.487171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:50771)
0619 02:35:19.487487 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:19.487511 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:19.487676 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:19.487993 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:19.487996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:19.488229 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:19.488338 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:19.488344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.489146 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.489150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:19.489754 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:19.489761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.490222 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.490225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:19.490312 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:19.490834 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:19.490846 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:19.492773 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":88,"threads_started":1}
W20260619 02:35:19.493217 17166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:19.487238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:50771 (local address 127.7.194.254:35519)
0619 02:35:19.487447 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:19.487452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:19.487620 (+ 168us) server_negotiation.cc:407] Connection header received
0619 02:35:19.487821 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:19.487826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:19.487883 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:19.487977 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:19.488455 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.489002 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.489871 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.490103 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.490217 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:19.492918 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:50771: BlockingRecv error: recv got EOF from 127.7.194.194:50771 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Jun 19 02:35:19 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:19 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:19.681977 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:19.675717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:39609)
0619 02:35:19.676051 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:19.676076 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:19.676222 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:19.676636 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:19.676640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:19.676864 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:19.676966 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:19.676972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.677968 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.677971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:19.678697 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:19.678706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.679135 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.679137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:19.679201 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:19.679784 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:19.679798 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:19.681804 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":88,"threads_started":1}
W20260619 02:35:19.682042 17166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:19.675861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:39609 (local address 127.7.194.254:35519)
0619 02:35:19.676030 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:19.676035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:19.676153 (+ 118us) server_negotiation.cc:407] Connection header received
0619 02:35:19.676447 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:19.676452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:19.676511 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:19.676613 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:19.677116 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.677843 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.678809 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:19.679030 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:19.679118 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:19.681930 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:39609: BlockingRecv error: recv got EOF from 127.7.194.195:39609 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 02:35:20 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:20 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:20.280519 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:20.274636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:45155)
0619 02:35:20.274958 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:20.274972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:20.275076 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:20.275346 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:20.275348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:20.275541 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:20.275638 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:20.275643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.276397 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.276399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:20.277124 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:20.277135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.277694 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.277697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:20.277787 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:20.278396 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:20.278409 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:20.280317 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
W20260619 02:35:20.280596 17170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:20.274800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:45155 (local address 127.7.194.254:35519)
0619 02:35:20.275045 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:20.275049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:20.275064 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:35:20.275175 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:20.275178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:20.275223 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:20.275306 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:20.275747 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.276290 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.277265 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.277538 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.277691 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:20.280464 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:45155: BlockingRecv error: recv got EOF from 127.7.194.193:45155 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":55,"threads_started":1}
Jun 19 02:35:20 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:20 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:20.499238 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:20.493753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:34319)
0619 02:35:20.494076 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:20.494090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:20.494188 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:20.494420 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:20.494422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:20.494605 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:20.494701 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:20.494706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.495413 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.495415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:20.495973 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:20.495980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.496493 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.496497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:20.496568 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:20.497077 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:20.497090 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:20.499053 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
W20260619 02:35:20.499276 17170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:20.493940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:34319 (local address 127.7.194.254:35519)
0619 02:35:20.494059 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:20.494062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:20.494140 (+ 78us) server_negotiation.cc:407] Connection header received
0619 02:35:20.494264 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:20.494267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:20.494307 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:20.494364 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:20.494802 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.495314 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.496098 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.496330 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.496418 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:20.499170 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:34319: BlockingRecv error: recv got EOF from 127.7.194.194:34319 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 02:35:20 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:20 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:20.689327 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:20.682992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:35509)
0619 02:35:20.683572 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:20.683587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:20.683743 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:20.684043 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:20.684046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:20.684398 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:20.684530 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:20.684536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.685316 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.685320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:20.685956 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:20.685964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.686438 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.686442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:20.686528 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:20.687089 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:20.687101 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:20.689132 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":120,"threads_started":1}
W20260619 02:35:20.689389 17170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:20.683174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:35509 (local address 127.7.194.254:35519)
0619 02:35:20.683388 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:20.683393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:20.683683 (+ 290us) server_negotiation.cc:407] Connection header received
0619 02:35:20.683859 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:20.683863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:20.683922 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:20.684016 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:20.684664 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.685197 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.686081 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:20.686321 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:20.686436 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:20.689270 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:35509: BlockingRecv error: recv got EOF from 127.7.194.195:35509 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 02:35:21 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:21 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:21.287734 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:21.281604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:50439)
0619 02:35:21.281901 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:21.281916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:21.282046 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:21.282584 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:21.282590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:21.282826 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:21.282981 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:21.282993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.283921 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.283924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:21.284537 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:21.284546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.284972 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.284974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:21.285036 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:21.285585 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:21.285598 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:21.287543 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
W20260619 02:35:21.287807 17174 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:21.281766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:50439 (local address 127.7.194.254:35519)
0619 02:35:21.282111 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:21.282114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:21.282129 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:35:21.282202 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:21.282205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:21.282258 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:21.282370 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:21.283114 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.283809 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.284642 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.284876 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.284955 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:21.287677 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:50439: BlockingRecv error: recv got EOF from 127.7.194.193:50439 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":92,"threads_started":1}
Jun 19 02:35:21 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:21 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:21.511049 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:21.500057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:43377)
0619 02:35:21.500395 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:21.500409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:21.500504 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:21.500803 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:21.500805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:21.501039 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:21.501141 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:21.501147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.502035 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.502038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:21.502662 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:21.502672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.503148 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.503151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:21.503218 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:21.508473 (+ 5255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:21.508489 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:21.510863 (+ 2374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
W20260619 02:35:21.511133 17174 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:21.500245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:43377 (local address 127.7.194.254:35519)
0619 02:35:21.500379 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:21.500382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:21.500472 (+ 90us) server_negotiation.cc:407] Connection header received
0619 02:35:21.500595 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:21.500597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:21.500640 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:21.500711 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:21.501245 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.501893 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.502821 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.503040 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.503119 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:21.511014 (+ 7895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:43377: BlockingRecv error: recv got EOF from 127.7.194.194:43377 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 02:35:21 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:21 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.195@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:21.696614 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:21.690420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.195:44707)
0619 02:35:21.690783 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:21.690797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:21.690901 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:21.691282 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:21.691293 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:21.691550 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:21.691680 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:21.691687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.692554 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.692557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:21.693135 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:21.693143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.693694 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.693698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:21.693773 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:21.694335 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:21.694356 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:21.696407 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
W20260619 02:35:21.696661 17174 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:21.690574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:44707 (local address 127.7.194.254:35519)
0619 02:35:21.690746 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:21.690752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:21.690882 (+ 130us) server_negotiation.cc:407] Connection header received
0619 02:35:21.691029 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:21.691032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:21.691098 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:21.691199 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:21.691817 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.692447 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.693266 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:21.693503 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:21.693688 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:21.696541 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.195:44707: BlockingRecv error: recv got EOF from 127.7.194.195:44707 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 02:35:22 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:22 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.193@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:22.294626 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:22.288870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.193:60821)
0619 02:35:22.289199 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:22.289214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:22.289309 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:22.289508 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:22.289510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:22.289713 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:22.289808 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:22.289813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.290560 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.290563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:22.291162 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:22.291171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.291595 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.291598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:22.291663 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:22.292260 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:22.292273 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:22.294435 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":85,"threads_started":1}
W20260619 02:35:22.294772 17178 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:22.288907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:60821 (local address 127.7.194.254:35519)
0619 02:35:22.289162 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:22.289166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:22.289285 (+ 119us) server_negotiation.cc:407] Connection header received
0619 02:35:22.289386 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:22.289388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:22.289430 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:22.289497 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:22.289904 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.290446 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.291262 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.291489 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.291562 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:22.294631 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.193:60821: BlockingRecv error: recv got EOF from 127.7.194.193:60821 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":62,"threads_started":1}
Jun 19 02:35:22 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:35:22 dist-test-slave-9tkz krb5kdc[16120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.194.194@KRBTEST.COM for oryx/127.7.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:35:22.517796 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:22.511996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:35519 (local address 127.7.194.194:47403)
0619 02:35:22.512267 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:22.512281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:22.512376 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:22.512626 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:22.512628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:22.512828 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:22.512923 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:22.512929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.513775 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.513779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:22.514497 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:22.514507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.514974 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.514978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:22.515064 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:22.515632 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:22.515644 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:22.517674 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.194.254:35519: Server oryx/127.7.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
W20260619 02:35:22.518044 17178 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:22.512130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:47403 (local address 127.7.194.254:35519)
0619 02:35:22.512276 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:22.512280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:22.512336 (+ 56us) server_negotiation.cc:407] Connection header received
0619 02:35:22.512468 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:22.512470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:22.512518 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:22.512597 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:22.513029 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.513647 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.514618 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.514856 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.514970 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:22.517884 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.194.194:47403: BlockingRecv error: recv got EOF from 127.7.194.194:47403 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:35:22.585111 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 16703
I20260619 02:35:22.592021 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 16855
I20260619 02:35:22.597947 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 16987
I20260619 02:35:22.603724 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 16633
2026-06-19T02:35:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18118 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T02:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:35:22Z Disabled control of system clock
I20260619 02:35:22.638185 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:43273
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:43273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:35:22.758437 17183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:22.758742 17183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:22.758805 17183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:22.762972 17183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:35:22.763044 17183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:22.763060 17183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:22.763078 17183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:35:22.763142 17183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:35:22.768153 17183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:43273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:43273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:22.769374 17183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:22.770890 17183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:22.779945 17191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:22.779913 17188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:22.779913 17189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:22.780376 17183 server_base.cc:1034] running on GCE node
I20260619 02:35:22.781193 17183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:22.782786 17183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:22.784005 17183 hybrid_clock.cc:648] HybridClock initialized: now 1781836522783988 us; error 55 us; skew 500 ppm
I20260619 02:35:22.786499 17183 webserver.cc:460] Webserver started at http://127.7.194.254:36825/ using document root <none> and password file <none>
I20260619 02:35:22.787326 17183 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:22.787417 17183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:22.787662 17183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:22.790630 17183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "7978da2b4fc24df09a6f48152dc93078"
format_stamp: "Formatted at 2026-06-19 02:35:22 on dist-test-slave-9tkz"
server_key: "10690692c107efe507f161558bb38360"
server_key_iv: "a2011eeda673d44072f8c50a72a4d6d8"
server_key_version: "encryptionkey@0"
I20260619 02:35:22.791404 17183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "7978da2b4fc24df09a6f48152dc93078"
format_stamp: "Formatted at 2026-06-19 02:35:22 on dist-test-slave-9tkz"
server_key: "10690692c107efe507f161558bb38360"
server_key_iv: "a2011eeda673d44072f8c50a72a4d6d8"
server_key_version: "encryptionkey@0"
I20260619 02:35:22.796955 17183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:35:22.800366 17197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:22.801836 17183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 02:35:22.801986 17183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "7978da2b4fc24df09a6f48152dc93078"
format_stamp: "Formatted at 2026-06-19 02:35:22 on dist-test-slave-9tkz"
server_key: "10690692c107efe507f161558bb38360"
server_key_iv: "a2011eeda673d44072f8c50a72a4d6d8"
server_key_version: "encryptionkey@0"
I20260619 02:35:22.802116 17183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:22.817121 17183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:22.818058 17183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:22.818290 17183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:22.825417 17183 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:43273
I20260619 02:35:22.825419 17249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:43273 every 8 connection(s)
I20260619 02:35:22.826524 17183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:35:22.829487 17250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:22.835255 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 17183
I20260619 02:35:22.835391 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:35:22.835714 7947 external_mini_cluster.cc:1428] Setting key 3a432cb8eb2dc5cf2ddb4b7fa199a94a
I20260619 02:35:22.835855 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078: Bootstrap starting.
I20260619 02:35:22.839218 17250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:22.840023 17250 log.cc:824] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:22.842176 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078: No bootstrap required, opened a new log
I20260619 02:35:22.845712 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:22.837054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.7.194.254:43273)
0619 02:35:22.837805 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:22.837823 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:22.837863 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:35:22.838573 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:22.838595 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:22.839035 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:22.839408 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:22.839871 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.840954 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.841977 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:22.842537 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:22.842714 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:22.843785 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:22.843790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:22.843803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:35:22.844228 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:22.844288 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:35:22.844308 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:22.844423 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:22.844584 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 02:35:22.844862 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260619 02:35:22.846143 17250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7978da2b4fc24df09a6f48152dc93078" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43273 } }
I20260619 02:35:22.846408 17250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:22.846490 17250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7978da2b4fc24df09a6f48152dc93078, State: Initialized, Role: FOLLOWER
I20260619 02:35:22.847034 17250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7978da2b4fc24df09a6f48152dc93078" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43273 } }
I20260619 02:35:22.847194 17250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:35:22.847246 17250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:35:22.847355 17250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:22.848590 17250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7978da2b4fc24df09a6f48152dc93078" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43273 } }
I20260619 02:35:22.848903 17250 leader_election.cc:302] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7978da2b4fc24df09a6f48152dc93078; no voters:
I20260619 02:35:22.849176 17250 leader_election.cc:288] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:35:22.849323 17255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:22.849601 17255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [term 1 LEADER]: Becoming Leader. State: Replica: 7978da2b4fc24df09a6f48152dc93078, State: Running, Role: LEADER
I20260619 02:35:22.849992 17255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7978da2b4fc24df09a6f48152dc93078" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43273 } }
I20260619 02:35:22.850533 17250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:35:22.851698 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7978da2b4fc24df09a6f48152dc93078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7978da2b4fc24df09a6f48152dc93078" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43273 } } }
I20260619 02:35:22.851830 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:22.852238 17264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:35:22.852805 17257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7978da2b4fc24df09a6f48152dc93078. Latest consensus state: current_term: 1 leader_uuid: "7978da2b4fc24df09a6f48152dc93078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7978da2b4fc24df09a6f48152dc93078" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 43273 } } }
I20260619 02:35:22.852897 17257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078 [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:22.855741 17264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:35:22.861853 17264 catalog_manager.cc:1347] Generated new cluster ID: be094bf9695447b0840283abf2f328d7
I20260619 02:35:22.861984 17264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:35:22.880455 17264 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:35:22.881899 17264 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:35:22.888693 17264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7978da2b4fc24df09a6f48152dc93078: Generated new TSK 0
I20260619 02:35:22.889659 17264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:35:22.898068 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:43273
--builtin_ntp_servers=127.7.194.212:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 02:35:23.017052 17274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:23.017345 17274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:23.017385 17274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:23.021653 17274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:23.021729 17274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:23.021793 17274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:35:23.026911 17274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:43273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:23.028107 17274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:23.029613 17274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:23.039851 17280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:23.039930 17274 server_base.cc:1034] running on GCE node
W20260619 02:35:23.039845 17282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:23.039851 17279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:23.040515 17274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:23.041235 17274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:23.042398 17274 hybrid_clock.cc:648] HybridClock initialized: now 1781836523042348 us; error 67 us; skew 500 ppm
I20260619 02:35:23.044348 17274 webserver.cc:460] Webserver started at http://127.7.194.193:45689/ using document root <none> and password file <none>
I20260619 02:35:23.044939 17274 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:23.044999 17274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:23.045208 17274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:23.047324 17274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "ea55892ee743474481ab04c27d703c6d"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "6f1fc47aea32c32bcdaaa17e75f45435"
server_key_iv: "41e41b6bc76ab85a2460571edae915e5"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.047883 17274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "ea55892ee743474481ab04c27d703c6d"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "6f1fc47aea32c32bcdaaa17e75f45435"
server_key_iv: "41e41b6bc76ab85a2460571edae915e5"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.051764 17274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:35:23.054644 17288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.055962 17274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:23.056059 17274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "ea55892ee743474481ab04c27d703c6d"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "6f1fc47aea32c32bcdaaa17e75f45435"
server_key_iv: "41e41b6bc76ab85a2460571edae915e5"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.056196 17274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:23.077504 17274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:23.078325 17274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:23.078537 17274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:23.079190 17274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:23.080188 17274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:23.080235 17274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.080279 17274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:23.080320 17274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.090075 17274 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:37321
I20260619 02:35:23.090096 17401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:37321 every 8 connection(s)
I20260619 02:35:23.091071 17274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:35:23.095027 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 17274
I20260619 02:35:23.095167 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:35:23.095501 7947 external_mini_cluster.cc:1428] Setting key 4535ee50c018e901e7808b545fde7e1f
I20260619 02:35:23.097747 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:43273
--builtin_ntp_servers=127.7.194.212:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:35:23.101996 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:23.093098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:52459 (local address 127.7.194.254:43273)
0619 02:35:23.093257 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:23.093262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:23.094556 (+ 1294us) server_negotiation.cc:407] Connection header received
0619 02:35:23.096410 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:23.096413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:23.096467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:23.096554 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:23.097763 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.098366 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.099832 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.100074 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.100201 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:23.101419 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:23.101423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:23.101427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:35:23.101445 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:23.101465 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:35:23.101473 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:23.101548 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:23.101776 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 02:35:23.101845 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:35:23.102949 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:23.093643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:43273 (local address 127.7.194.193:52459)
0619 02:35:23.094397 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:23.094441 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:23.096181 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:23.096757 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:23.096765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:23.096858 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:23.097582 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:23.097608 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.098500 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.098507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:23.099695 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:23.099705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.100236 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.100240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:23.100415 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:23.101123 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:35:23.101125 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:23.101190 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:35:23.101261 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:23.101620 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:23.101629 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:23.101635 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:35:23.101778 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 02:35:23.102106 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":112,"threads_started":1}
I20260619 02:35:23.104564 17402 heartbeater.cc:347] Connected to a master server at 127.7.194.254:43273
I20260619 02:35:23.104831 17402 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:23.105417 17402 heartbeater.cc:511] Master 127.7.194.254:43273 requested a full tablet report, sending...
I20260619 02:35:23.107323 17214 ts_manager.cc:194] Registered new tserver with Master: ea55892ee743474481ab04c27d703c6d (127.7.194.193:37321)
I20260619 02:35:23.109093 17214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.193:52459
W20260619 02:35:23.214520 17405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:23.214821 17405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:23.214860 17405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:23.219286 17405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:23.219358 17405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:23.219419 17405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:35:23.224366 17405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:43273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:23.225531 17405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:23.227114 17405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:23.237690 17410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:23.237648 17411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:23.237648 17413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:23.237942 17405 server_base.cc:1034] running on GCE node
I20260619 02:35:23.238430 17405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:23.239121 17405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:23.240309 17405 hybrid_clock.cc:648] HybridClock initialized: now 1781836523240295 us; error 32 us; skew 500 ppm
I20260619 02:35:23.242314 17405 webserver.cc:460] Webserver started at http://127.7.194.194:36005/ using document root <none> and password file <none>
I20260619 02:35:23.242924 17405 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:23.243016 17405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:23.243264 17405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:23.245430 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "1b091a855e2f48fc8bf44d44e6e38cbc"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "d0cf43af0fcfde9d31970135804cbc31"
server_key_iv: "c7cd3dff63de25745759580d01c717c0"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.246117 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "1b091a855e2f48fc8bf44d44e6e38cbc"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "d0cf43af0fcfde9d31970135804cbc31"
server_key_iv: "c7cd3dff63de25745759580d01c717c0"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.250033 17405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:35:23.252974 17419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.254390 17405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:23.254518 17405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "1b091a855e2f48fc8bf44d44e6e38cbc"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "d0cf43af0fcfde9d31970135804cbc31"
server_key_iv: "c7cd3dff63de25745759580d01c717c0"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.254642 17405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:23.265786 17405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:23.266518 17405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:23.266724 17405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:23.267369 17405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:23.268373 17405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:23.268426 17405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.268496 17405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:23.268536 17405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.277345 17405 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:44397
I20260619 02:35:23.277366 17532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:44397 every 8 connection(s)
I20260619 02:35:23.278360 17405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:35:23.284642 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 17405
I20260619 02:35:23.284796 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:35:23.285152 7947 external_mini_cluster.cc:1428] Setting key fae5698525e5f4b71bbd2b1faa66961b
I20260619 02:35:23.287283 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:43273
--builtin_ntp_servers=127.7.194.212:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:35:23.288689 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:23.280331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:41873 (local address 127.7.194.254:43273)
0619 02:35:23.280544 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:23.280548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:23.281777 (+ 1229us) server_negotiation.cc:407] Connection header received
0619 02:35:23.283250 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:23.283253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:23.283311 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:23.283392 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:23.284417 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.285069 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.286512 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.286802 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.286922 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:23.288086 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:23.288093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:23.288099 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:35:23.288121 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:23.288149 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:35:23.288163 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:23.288247 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:23.288396 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 02:35:23.288471 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 02:35:23.289884 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:23.280782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:43273 (local address 127.7.194.194:41873)
0619 02:35:23.281527 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:23.281658 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:23.283011 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:23.283581 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:23.283589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:23.283682 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:23.284241 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:23.284255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.285200 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.285206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:23.286364 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:23.286374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.286959 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.286962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:23.287108 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:23.287811 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:35:23.287812 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:23.287857 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:35:23.287927 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:23.288258 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:23.288265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:23.288269 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:35:23.288371 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:35:23.288686 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260619 02:35:23.291167 17533 heartbeater.cc:347] Connected to a master server at 127.7.194.254:43273
I20260619 02:35:23.291443 17533 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:23.292179 17533 heartbeater.cc:511] Master 127.7.194.254:43273 requested a full tablet report, sending...
I20260619 02:35:23.293426 17214 ts_manager.cc:194] Registered new tserver with Master: 1b091a855e2f48fc8bf44d44e6e38cbc (127.7.194.194:44397)
I20260619 02:35:23.294049 17214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.194:41873
W20260619 02:35:23.404706 17536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:23.405000 17536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:23.405038 17536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:23.409384 17536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:23.409458 17536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:23.409579 17536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:35:23.414552 17536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:43273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:23.415745 17536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:23.417214 17536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:23.427131 17542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:23.427131 17541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:23.427225 17544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:23.427287 17536 server_base.cc:1034] running on GCE node
I20260619 02:35:23.427868 17536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:23.428579 17536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:23.429729 17536 hybrid_clock.cc:648] HybridClock initialized: now 1781836523429697 us; error 41 us; skew 500 ppm
I20260619 02:35:23.431659 17536 webserver.cc:460] Webserver started at http://127.7.194.195:33029/ using document root <none> and password file <none>
I20260619 02:35:23.432233 17536 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:23.432294 17536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:23.432461 17536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:23.434657 17536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "db325ccab3c242c2bd1ac97cc8591ce5"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "9879aeb59a31988c7eaec6f06e7c7967"
server_key_iv: "1f571094986f7224ef49f78040962d94"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.435186 17536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "db325ccab3c242c2bd1ac97cc8591ce5"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "9879aeb59a31988c7eaec6f06e7c7967"
server_key_iv: "1f571094986f7224ef49f78040962d94"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.438963 17536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:35:23.441641 17550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.442942 17536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:35:23.443085 17536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "db325ccab3c242c2bd1ac97cc8591ce5"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "9879aeb59a31988c7eaec6f06e7c7967"
server_key_iv: "1f571094986f7224ef49f78040962d94"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.443209 17536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:23.459045 17536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:23.459945 17536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:23.460147 17536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:23.460829 17536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:23.461946 17536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:23.462016 17536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.462090 17536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:23.462133 17536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.471423 17536 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:32819
I20260619 02:35:23.471442 17663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:32819 every 8 connection(s)
I20260619 02:35:23.472405 17536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:35:23.474820 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 17536
I20260619 02:35:23.474910 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:35:23.475248 7947 external_mini_cluster.cc:1428] Setting key b253849fb01bb2a65484ecda4456534d
I20260619 02:35:23.482331 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:23.474414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:34787 (local address 127.7.194.254:43273)
0619 02:35:23.474554 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:23.474558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:23.475851 (+ 1293us) server_negotiation.cc:407] Connection header received
0619 02:35:23.477688 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:23.477691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:23.477743 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:23.477811 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:23.478732 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.479243 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.480345 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.480551 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.480625 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:23.481773 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:23.481780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:23.481786 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:35:23.481804 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:23.481829 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:35:23.481843 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:23.481927 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:23.482070 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 02:35:23.482142 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:35:23.483127 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:23.474885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:43273 (local address 127.7.194.195:34787)
0619 02:35:23.475683 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:23.475752 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:23.477407 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:23.477984 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:23.477991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:23.478081 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:23.478597 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:23.478610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.479345 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.479347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:23.480241 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:23.480249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:23.480674 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:23.480677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:23.480832 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:23.481406 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:35:23.481408 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:23.481449 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:35:23.481514 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:23.481930 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:23.481937 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:23.481941 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:35:23.482037 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:35:23.482263 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":119,"threads_started":1}
I20260619 02:35:23.484447 17664 heartbeater.cc:347] Connected to a master server at 127.7.194.254:43273
I20260619 02:35:23.484723 17664 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:23.485332 17664 heartbeater.cc:511] Master 127.7.194.254:43273 requested a full tablet report, sending...
I20260619 02:35:23.486487 17214 ts_manager.cc:194] Registered new tserver with Master: db325ccab3c242c2bd1ac97cc8591ce5 (127.7.194.195:32819)
I20260619 02:35:23.487079 17214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.195:34787
I20260619 02:35:23.489334 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:35:23.492081 17253 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:23.491243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.7.194.254:43273)
0619 02:35:23.491374 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:23.491378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:23.491563 (+ 185us) server_negotiation.cc:407] Connection header received
0619 02:35:23.491660 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:23.491662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:23.491712 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:23.491776 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:23.491982 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50878: BlockingRecv error: recv got EOF from 127.0.0.1:50878 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:35:23.492308 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 17274
I20260619 02:35:23.498183 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 17405
I20260619 02:35:23.503995 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 17536
I20260619 02:35:23.509763 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 17183
2026-06-19T02:35:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (902 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T02:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:35:23Z Disabled control of system clock
I20260619 02:35:23.540031 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:41407
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:41407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:35:23.655395 17675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:23.655692 17675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:23.655733 17675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:23.659765 17675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:35:23.659829 17675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:23.659845 17675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:23.659862 17675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:35:23.659878 17675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:35:23.664744 17675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:41407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:41407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:23.666006 17675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:23.667428 17675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:23.676313 17681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:23.676378 17683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:23.676343 17675 server_base.cc:1034] running on GCE node
W20260619 02:35:23.676395 17680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:23.677289 17675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:23.678647 17675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:23.679876 17675 hybrid_clock.cc:648] HybridClock initialized: now 1781836523679845 us; error 54 us; skew 500 ppm
I20260619 02:35:23.682106 17675 webserver.cc:460] Webserver started at http://127.7.194.254:34003/ using document root <none> and password file <none>
I20260619 02:35:23.682734 17675 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:23.682830 17675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:23.683106 17675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:23.685276 17675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "f27bf6dc5bfc4530826d2d116e8caeb5"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "de311da89bf06d487dc7488b7c32507e"
server_key_iv: "ee448bc3b4407e0ad37ccb5c3e87f710"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.685976 17675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "f27bf6dc5bfc4530826d2d116e8caeb5"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "de311da89bf06d487dc7488b7c32507e"
server_key_iv: "ee448bc3b4407e0ad37ccb5c3e87f710"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.689877 17675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:35:23.692590 17689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.694079 17675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:23.694239 17675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "f27bf6dc5bfc4530826d2d116e8caeb5"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "de311da89bf06d487dc7488b7c32507e"
server_key_iv: "ee448bc3b4407e0ad37ccb5c3e87f710"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.694366 17675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:23.699229 17675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:23.699867 17675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:23.700057 17675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:23.707186 17675 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:41407
I20260619 02:35:23.707216 17741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:41407 every 8 connection(s)
I20260619 02:35:23.708181 17675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:35:23.711721 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:23.716238 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 17675
I20260619 02:35:23.716360 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:35:23.716662 7947 external_mini_cluster.cc:1428] Setting key f41b3782b1da476257ed62a156187a54
I20260619 02:35:23.718346 17742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5: Bootstrap starting.
I20260619 02:35:23.720762 17742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:23.721611 17742 log.cc:824] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:23.722750 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:23.718234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.7.194.254:41407)
0619 02:35:23.718946 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:23.718962 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:23.718998 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:35:23.719672 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:23.719689 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:23.720069 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:23.720441 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:23.721258 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:23.721264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:23.721276 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:35:23.721286 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:23.721342 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:35:23.721358 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:23.721476 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:23.721601 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 02:35:23.721865 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":151,"threads_started":1}
I20260619 02:35:23.723766 17742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5: No bootstrap required, opened a new log
I20260619 02:35:23.727421 17742 raft_consensus.cc:348] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27bf6dc5bfc4530826d2d116e8caeb5" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41407 } }
I20260619 02:35:23.727646 17742 raft_consensus.cc:374] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:23.727717 17742 raft_consensus.cc:729] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27bf6dc5bfc4530826d2d116e8caeb5, State: Initialized, Role: FOLLOWER
I20260619 02:35:23.728178 17742 consensus_queue.cc:260] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27bf6dc5bfc4530826d2d116e8caeb5" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41407 } }
I20260619 02:35:23.728304 17742 raft_consensus.cc:388] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:35:23.728385 17742 raft_consensus.cc:482] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:35:23.728480 17742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:23.729487 17742 raft_consensus.cc:504] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27bf6dc5bfc4530826d2d116e8caeb5" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41407 } }
I20260619 02:35:23.729872 17742 leader_election.cc:302] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f27bf6dc5bfc4530826d2d116e8caeb5; no voters:
I20260619 02:35:23.730217 17742 leader_election.cc:288] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:35:23.730325 17747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:23.730568 17747 raft_consensus.cc:686] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [term 1 LEADER]: Becoming Leader. State: Replica: f27bf6dc5bfc4530826d2d116e8caeb5, State: Running, Role: LEADER
I20260619 02:35:23.730968 17747 consensus_queue.cc:237] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27bf6dc5bfc4530826d2d116e8caeb5" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41407 } }
I20260619 02:35:23.731489 17742 sys_catalog.cc:564] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:35:23.732745 17748 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f27bf6dc5bfc4530826d2d116e8caeb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27bf6dc5bfc4530826d2d116e8caeb5" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41407 } } }
I20260619 02:35:23.732883 17748 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:23.733289 17756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:35:23.733237 17749 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f27bf6dc5bfc4530826d2d116e8caeb5. Latest consensus state: current_term: 1 leader_uuid: "f27bf6dc5bfc4530826d2d116e8caeb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27bf6dc5bfc4530826d2d116e8caeb5" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41407 } } }
I20260619 02:35:23.733497 17749 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:23.736625 17756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:35:23.742648 17756 catalog_manager.cc:1347] Generated new cluster ID: 5d9b124a20744657bf859829c19d1460
I20260619 02:35:23.742734 17756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:35:23.761444 17756 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:35:23.761847 17756 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:35:23.770246 17756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f27bf6dc5bfc4530826d2d116e8caeb5: Generated new TSK 0
I20260619 02:35:23.771035 17756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:35:23.777061 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:41407
--builtin_ntp_servers=127.7.194.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:35:23.916463 17766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:23.916754 17766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:23.916797 17766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:23.921051 17766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:23.921123 17766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:23.921182 17766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:35:23.926157 17766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:41407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:23.927281 17766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:23.928700 17766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:23.939666 17772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:23.939653 17774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:23.939654 17771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:23.939894 17766 server_base.cc:1034] running on GCE node
I20260619 02:35:23.940390 17766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:23.941071 17766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:23.942257 17766 hybrid_clock.cc:648] HybridClock initialized: now 1781836523942219 us; error 52 us; skew 500 ppm
I20260619 02:35:23.944159 17766 webserver.cc:460] Webserver started at http://127.7.194.193:34791/ using document root <none> and password file <none>
I20260619 02:35:23.944769 17766 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:23.944833 17766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:23.945082 17766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:23.947307 17766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "8fa549806cf649adaa3f9bec2f298ac4"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "72ded3de2ce812162ca948e92e1c9e43"
server_key_iv: "26e5bff4eb08148e4aaee3717943d9a3"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.947880 17766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "8fa549806cf649adaa3f9bec2f298ac4"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "72ded3de2ce812162ca948e92e1c9e43"
server_key_iv: "26e5bff4eb08148e4aaee3717943d9a3"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.952030 17766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:35:23.954695 17780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.956117 17766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:35:23.956278 17766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "8fa549806cf649adaa3f9bec2f298ac4"
format_stamp: "Formatted at 2026-06-19 02:35:23 on dist-test-slave-9tkz"
server_key: "72ded3de2ce812162ca948e92e1c9e43"
server_key_iv: "26e5bff4eb08148e4aaee3717943d9a3"
server_key_version: "encryptionkey@0"
I20260619 02:35:23.956419 17766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:23.961159 17766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:23.961876 17766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:23.962064 17766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:23.962606 17766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:23.963532 17766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:23.963600 17766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.963670 17766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:23.963723 17766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:23.972568 17766 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:39021
I20260619 02:35:23.972587 17893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:39021 every 8 connection(s)
I20260619 02:35:23.973531 17766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:35:23.974406 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 17766
I20260619 02:35:23.974530 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:35:23.974927 7947 external_mini_cluster.cc:1428] Setting key 58f4f9f406c2383c068362c30436b469
I20260619 02:35:23.977182 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:41407
--builtin_ntp_servers=127.7.194.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 02:35:23.981158 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:23.975815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:38147 (local address 127.7.194.254:41407)
0619 02:35:23.975988 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:23.975993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:23.977219 (+ 1226us) server_negotiation.cc:407] Connection header received
0619 02:35:23.979225 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:23.979231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:23.979281 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:23.979383 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:23.980597 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:23.980604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:23.980610 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:35:23.980629 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:23.980657 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:35:23.980670 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:23.980775 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:23.980899 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:35:23.980952 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:35:23.982040 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:23.976171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:41407 (local address 127.7.194.193:38147)
0619 02:35:23.977048 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:23.977110 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:23.978996 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:23.979558 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:23.979565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:23.979661 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:23.980311 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:35:23.980313 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:23.980356 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:35:23.980428 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:23.980762 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:23.980768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:23.980773 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:35:23.980908 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 02:35:23.981108 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":150,"threads_started":1}
I20260619 02:35:23.983367 17894 heartbeater.cc:347] Connected to a master server at 127.7.194.254:41407
I20260619 02:35:23.983599 17894 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:23.984027 17894 heartbeater.cc:511] Master 127.7.194.254:41407 requested a full tablet report, sending...
I20260619 02:35:23.985826 17706 ts_manager.cc:194] Registered new tserver with Master: 8fa549806cf649adaa3f9bec2f298ac4 (127.7.194.193:39021)
W20260619 02:35:24.094151 17897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:24.094439 17897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:24.094480 17897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:24.098606 17897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:24.098672 17897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:24.098732 17897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:35:24.103695 17897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:41407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:24.104841 17897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:24.106272 17897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:24.116029 17902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:24.116041 17903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:24.116117 17905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:24.116609 17897 server_base.cc:1034] running on GCE node
I20260619 02:35:24.117027 17897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:24.117744 17897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:24.118930 17897 hybrid_clock.cc:648] HybridClock initialized: now 1781836524118913 us; error 28 us; skew 500 ppm
I20260619 02:35:24.121361 17897 webserver.cc:460] Webserver started at http://127.7.194.194:32837/ using document root <none> and password file <none>
I20260619 02:35:24.122077 17897 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:24.122148 17897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:24.122390 17897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:24.124544 17897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "a3d7312b3023432fb7cf3d820c712bf6"
format_stamp: "Formatted at 2026-06-19 02:35:24 on dist-test-slave-9tkz"
server_key: "85eb6ee8f18a05c6c4eddcdb50e58912"
server_key_iv: "61699fb42ad98069efd8a4d6a0fe5120"
server_key_version: "encryptionkey@0"
I20260619 02:35:24.125142 17897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "a3d7312b3023432fb7cf3d820c712bf6"
format_stamp: "Formatted at 2026-06-19 02:35:24 on dist-test-slave-9tkz"
server_key: "85eb6ee8f18a05c6c4eddcdb50e58912"
server_key_iv: "61699fb42ad98069efd8a4d6a0fe5120"
server_key_version: "encryptionkey@0"
I20260619 02:35:24.129115 17897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:35:24.131964 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:24.133442 17897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:35:24.133632 17897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "a3d7312b3023432fb7cf3d820c712bf6"
format_stamp: "Formatted at 2026-06-19 02:35:24 on dist-test-slave-9tkz"
server_key: "85eb6ee8f18a05c6c4eddcdb50e58912"
server_key_iv: "61699fb42ad98069efd8a4d6a0fe5120"
server_key_version: "encryptionkey@0"
I20260619 02:35:24.133771 17897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:24.138440 17897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:24.139042 17897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:24.139181 17897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:24.139680 17897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:24.140591 17897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:24.140636 17897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:24.140674 17897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:24.140689 17897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:24.149490 17897 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:35823
I20260619 02:35:24.149504 18024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:35823 every 8 connection(s)
I20260619 02:35:24.150521 17897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:35:24.154543 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 17897
I20260619 02:35:24.154693 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:35:24.154984 7947 external_mini_cluster.cc:1428] Setting key afc144c2dba02feceec7f6f17acfa338
I20260619 02:35:24.157181 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:24.152436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:60835 (local address 127.7.194.254:41407)
0619 02:35:24.152589 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:24.152594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:24.153840 (+ 1246us) server_negotiation.cc:407] Connection header received
0619 02:35:24.155289 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:24.155293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:24.155339 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:24.155443 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:24.156682 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:24.156685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:24.156687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:35:24.156698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:24.156717 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:35:24.156725 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:24.156797 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:24.157011 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 02:35:24.157059 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:35:24.157212 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:41407
--builtin_ntp_servers=127.7.194.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 02:35:24.158387 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:24.152932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:41407 (local address 127.7.194.194:60835)
0619 02:35:24.153687 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:24.153729 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:24.155033 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:24.155655 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:24.155667 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:24.155770 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:24.156375 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:35:24.156376 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:24.156419 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:35:24.156526 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:24.156813 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:24.156821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:24.156827 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:35:24.157015 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 02:35:24.157283 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":125,"threads_started":1}
I20260619 02:35:24.159873 18025 heartbeater.cc:347] Connected to a master server at 127.7.194.254:41407
I20260619 02:35:24.160111 18025 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:24.160553 18025 heartbeater.cc:511] Master 127.7.194.254:41407 requested a full tablet report, sending...
I20260619 02:35:24.161856 17706 ts_manager.cc:194] Registered new tserver with Master: a3d7312b3023432fb7cf3d820c712bf6 (127.7.194.194:35823)
W20260619 02:35:24.275741 18028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:24.276021 18028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:24.276062 18028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:24.280344 18028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:24.280416 18028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:24.280509 18028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:35:24.285671 18028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:41407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:24.286859 18028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:24.288363 18028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:24.298409 18034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:24.298384 18033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:24.298378 18036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:24.298818 18028 server_base.cc:1034] running on GCE node
I20260619 02:35:24.299263 18028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:24.300051 18028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:24.301273 18028 hybrid_clock.cc:648] HybridClock initialized: now 1781836524301261 us; error 31 us; skew 500 ppm
I20260619 02:35:24.303297 18028 webserver.cc:460] Webserver started at http://127.7.194.195:46777/ using document root <none> and password file <none>
I20260619 02:35:24.303990 18028 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:24.304057 18028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:24.304282 18028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:24.306486 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "79d09cdb34ce448483171d2e7cb9969b"
format_stamp: "Formatted at 2026-06-19 02:35:24 on dist-test-slave-9tkz"
server_key: "295538d11cbd682823107d1778ea325f"
server_key_iv: "088a2afcd873af0fb5bb2d70a6bda368"
server_key_version: "encryptionkey@0"
I20260619 02:35:24.307073 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "79d09cdb34ce448483171d2e7cb9969b"
format_stamp: "Formatted at 2026-06-19 02:35:24 on dist-test-slave-9tkz"
server_key: "295538d11cbd682823107d1778ea325f"
server_key_iv: "088a2afcd873af0fb5bb2d70a6bda368"
server_key_version: "encryptionkey@0"
I20260619 02:35:24.310827 18028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:35:24.313436 18042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:24.314836 18028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:35:24.314980 18028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "79d09cdb34ce448483171d2e7cb9969b"
format_stamp: "Formatted at 2026-06-19 02:35:24 on dist-test-slave-9tkz"
server_key: "295538d11cbd682823107d1778ea325f"
server_key_iv: "088a2afcd873af0fb5bb2d70a6bda368"
server_key_version: "encryptionkey@0"
I20260619 02:35:24.315102 18028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:24.319823 18028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:24.320439 18028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:24.320622 18028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:24.321173 18028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:24.322142 18028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:24.322209 18028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:24.322274 18028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:24.322322 18028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:24.331281 18028 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:40059
I20260619 02:35:24.331326 18155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:40059 every 8 connection(s)
I20260619 02:35:24.332362 18028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:35:24.335427 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 18028
I20260619 02:35:24.335534 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:35:24.335866 7947 external_mini_cluster.cc:1428] Setting key 037f12fb36974202093a573d52c01875
I20260619 02:35:24.339483 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:24.334442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:51763 (local address 127.7.194.254:41407)
0619 02:35:24.334607 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:24.334612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:24.335867 (+ 1255us) server_negotiation.cc:407] Connection header received
0619 02:35:24.337462 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:24.337466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:24.337512 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:24.337689 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:24.339008 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:24.339011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:24.339013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:35:24.339026 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:24.339049 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:35:24.339059 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:24.339147 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:24.339335 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 02:35:24.339377 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:35:24.340431 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:24.334827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:41407 (local address 127.7.194.195:51763)
0619 02:35:24.335688 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:24.335745 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:24.337260 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:24.337842 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:24.337851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:24.337971 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:24.338687 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:35:24.338688 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:24.338732 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:35:24.338802 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:24.339171 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:24.339179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:24.339184 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:35:24.339334 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 02:35:24.339534 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":114,"threads_started":1}
I20260619 02:35:24.341497 18156 heartbeater.cc:347] Connected to a master server at 127.7.194.254:41407
I20260619 02:35:24.341777 18156 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:24.342231 18156 heartbeater.cc:511] Master 127.7.194.254:41407 requested a full tablet report, sending...
I20260619 02:35:24.343242 17706 ts_manager.cc:194] Registered new tserver with Master: 79d09cdb34ce448483171d2e7cb9969b (127.7.194.195:40059)
I20260619 02:35:24.350175 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:35:24.353274 17745 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:35:24.352240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.7.194.254:41407)
0619 02:35:24.352414 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:24.352420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:24.352546 (+ 126us) server_negotiation.cc:407] Connection header received
0619 02:35:24.352682 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:24.352685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:24.352731 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:24.352809 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:24.353130 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51108: BlockingRecv error: recv got EOF from 127.0.0.1:51108 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 02:35:24.353372 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 17766
I20260619 02:35:24.359140 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 17897
I20260619 02:35:24.364439 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 18028
I20260619 02:35:24.369741 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 17675
2026-06-19T02:35:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (859 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:35:24 dist-test-slave-9tkz krb5kdc[18166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:35:24 dist-test-slave-9tkz krb5kdc[18166](info): set up 2 sockets
Jun 19 02:35:24 dist-test-slave-9tkz krb5kdc[18166](info): commencing operation
krb5kdc: starting...
W20260619 02:35:25.406661 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:35:25 dist-test-slave-9tkz krb5kdc[18166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:35:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:35:25.533459 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38209
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:25.649039 18179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:25.649358 18179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:25.649430 18179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:25.653635 18179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:35:25.653728 18179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:25.653765 18179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:25.653805 18179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:35:25.653839 18179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:35:25.659432 18179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:25.660601 18179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:25.661772 18179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:25.670413 18185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:25.670369 18184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:25.670378 18187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:25.670643 18179 server_base.cc:1034] running on GCE node
I20260619 02:35:25.671377 18179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:25.672562 18179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:25.673754 18179 hybrid_clock.cc:648] HybridClock initialized: now 1781836525673736 us; error 34 us; skew 500 ppm
Jun 19 02:35:25 dist-test-slave-9tkz krb5kdc[18166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836525, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:25.676595 18179 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:35:25.677595 18179 webserver.cc:460] Webserver started at http://127.7.194.254:39401/ using document root <none> and password file <none>
I20260619 02:35:25.678198 18179 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:25.678288 18179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:25.678550 18179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:25.680680 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "0fb60e09afe343888d4266171824e31f"
format_stamp: "Formatted at 2026-06-19 02:35:25 on dist-test-slave-9tkz"
server_key: "4468f65f25f44f75bcfa49bde1639873"
server_key_iv: "5aebbfcaa066e16ea00337a314c1d0f8"
server_key_version: "encryptionkey@0"
I20260619 02:35:25.681310 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "0fb60e09afe343888d4266171824e31f"
format_stamp: "Formatted at 2026-06-19 02:35:25 on dist-test-slave-9tkz"
server_key: "4468f65f25f44f75bcfa49bde1639873"
server_key_iv: "5aebbfcaa066e16ea00337a314c1d0f8"
server_key_version: "encryptionkey@0"
I20260619 02:35:25.685194 18179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:35:25.687927 18194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:25.689203 18179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:35:25.689363 18179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "0fb60e09afe343888d4266171824e31f"
format_stamp: "Formatted at 2026-06-19 02:35:25 on dist-test-slave-9tkz"
server_key: "4468f65f25f44f75bcfa49bde1639873"
server_key_iv: "5aebbfcaa066e16ea00337a314c1d0f8"
server_key_version: "encryptionkey@0"
I20260619 02:35:25.689486 18179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:25.707906 18179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:25.711259 18179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:25.711479 18179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:25.718670 18179 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:38209
I20260619 02:35:25.718962 18246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:38209 every 8 connection(s)
I20260619 02:35:25.719817 18179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:35:25.720278 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 18179
I20260619 02:35:25.720409 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:35:25.720700 7947 external_mini_cluster.cc:1428] Setting key 6e42dc750fde655f96d06397cb49b259
I20260619 02:35:25.723470 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:25.733103 18247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f: Bootstrap starting.
Jun 19 02:35:25 dist-test-slave-9tkz krb5kdc[18166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:35:25.736987 18247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:25.738147 18247 log.cc:824] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:25.740674 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:25.722348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.7.194.254:38209)
0619 02:35:25.722863 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:25.722889 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:25.722927 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:35:25.723849 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:25.723868 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:25.724321 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:25.724807 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:25.726259 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:25.728100 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:25.729531 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:25.730392 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:25.730587 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:25.734182 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:25.734188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:25.734226 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:25.734266 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:25.736838 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:25.737386 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:25.737393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:25.737409 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:25.737590 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:25.737919 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:25.737923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:25.737925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:25.738585 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:25.738790 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:25.739121 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 02:35:25.739637 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":337,"thread_start_us":174,"threads_started":1}
I20260619 02:35:25.741295 18247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f: No bootstrap required, opened a new log
I20260619 02:35:25.747496 18247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb60e09afe343888d4266171824e31f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38209 } }
I20260619 02:35:25.747854 18247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:25.747934 18247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb60e09afe343888d4266171824e31f, State: Initialized, Role: FOLLOWER
I20260619 02:35:25.748543 18247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb60e09afe343888d4266171824e31f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38209 } }
I20260619 02:35:25.748723 18247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:35:25.748796 18247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:35:25.748916 18247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:25.750330 18247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb60e09afe343888d4266171824e31f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38209 } }
I20260619 02:35:25.750741 18247 leader_election.cc:302] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fb60e09afe343888d4266171824e31f; no voters:
I20260619 02:35:25.751140 18247 leader_election.cc:288] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:35:25.751276 18252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:25.751516 18252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [term 1 LEADER]: Becoming Leader. State: Replica: 0fb60e09afe343888d4266171824e31f, State: Running, Role: LEADER
I20260619 02:35:25.751828 18252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb60e09afe343888d4266171824e31f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38209 } }
I20260619 02:35:25.752485 18247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:35:25.753592 18253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fb60e09afe343888d4266171824e31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb60e09afe343888d4266171824e31f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38209 } } }
I20260619 02:35:25.753715 18253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:25.754101 18254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fb60e09afe343888d4266171824e31f. Latest consensus state: current_term: 1 leader_uuid: "0fb60e09afe343888d4266171824e31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb60e09afe343888d4266171824e31f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38209 } } }
I20260619 02:35:25.754258 18254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f [sys.catalog]: This master's current role is: LEADER
I20260619 02:35:25.755118 18261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:35:25.758701 18261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:35:25.766765 18261 catalog_manager.cc:1347] Generated new cluster ID: 37a036ec476c435ba76539c2b343345b
I20260619 02:35:25.766893 18261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:35:25.783835 18261 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:35:25.784830 18261 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:35:25.795992 18261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f: Generated new TSK 0
I20260619 02:35:25.796789 18261 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:35:25.828953 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38209
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:25.944917 18273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:25.945271 18273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:25.945350 18273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:25.949589 18273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:25.949702 18273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:25.949802 18273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:35:25.955663 18273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:25.956874 18273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:25.958135 18273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:25.968511 18278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:25.968509 18279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:25.968669 18273 server_base.cc:1034] running on GCE node
W20260619 02:35:25.968920 18281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:25.969383 18273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:25.970083 18273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:25.971280 18273 hybrid_clock.cc:648] HybridClock initialized: now 1781836525971234 us; error 50 us; skew 500 ppm
Jun 19 02:35:25 dist-test-slave-9tkz krb5kdc[18166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836525, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:25.974331 18273 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:35:25.975339 18273 webserver.cc:460] Webserver started at http://127.7.194.193:41883/ using document root <none> and password file <none>
I20260619 02:35:25.975970 18273 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:25.976071 18273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:25.976341 18273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:25.978648 18273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "49aabad8310a4761a04d92c39fb50c99"
format_stamp: "Formatted at 2026-06-19 02:35:25 on dist-test-slave-9tkz"
server_key: "c5411809098e48844dc82b75bdeabe1a"
server_key_iv: "2f544dba458d9d9870db70f09f9d6b64"
server_key_version: "encryptionkey@0"
I20260619 02:35:25.979300 18273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "49aabad8310a4761a04d92c39fb50c99"
format_stamp: "Formatted at 2026-06-19 02:35:25 on dist-test-slave-9tkz"
server_key: "c5411809098e48844dc82b75bdeabe1a"
server_key_iv: "2f544dba458d9d9870db70f09f9d6b64"
server_key_version: "encryptionkey@0"
I20260619 02:35:25.983547 18273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:35:25.986271 18288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:25.987632 18273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:35:25.987804 18273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "49aabad8310a4761a04d92c39fb50c99"
format_stamp: "Formatted at 2026-06-19 02:35:25 on dist-test-slave-9tkz"
server_key: "c5411809098e48844dc82b75bdeabe1a"
server_key_iv: "2f544dba458d9d9870db70f09f9d6b64"
server_key_version: "encryptionkey@0"
I20260619 02:35:25.987939 18273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:26.016531 18273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:26.020149 18273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:26.020402 18273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:26.021075 18273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:26.022081 18273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:26.022157 18273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:26.022233 18273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:26.022331 18273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:26.031409 18273 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:38177
I20260619 02:35:26.031425 18401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:38177 every 8 connection(s)
I20260619 02:35:26.032426 18273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:35:26.036309 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 18273
I20260619 02:35:26.036422 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:35:26.036713 7947 external_mini_cluster.cc:1428] Setting key ef6b322323a462ae67e2015f97c09430
Jun 19 02:35:26 dist-test-slave-9tkz krb5kdc[18166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836525, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:35:26.047597 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.034369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:58129 (local address 127.7.194.254:38209)
0619 02:35:26.034555 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.034564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.035485 (+ 921us) server_negotiation.cc:407] Connection header received
0619 02:35:26.036977 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.036980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.037042 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.037145 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:26.038732 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.039269 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.040444 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.040682 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.040793 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.043479 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:26.043483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:26.043486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:26.043522 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:26.045294 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:26.045898 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:26.045902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:26.045904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:26.046002 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:26.046356 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:26.046362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:26.046366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:26.046650 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:26.046762 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.047346 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.047464 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:35:26.048470 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.034843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38209 (local address 127.7.194.193:58129)
0619 02:35:26.035313 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:26.035360 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:26.036714 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:26.037517 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:26.037531 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:26.038037 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:26.038580 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.038594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.039398 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.039402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.040313 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.040322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.040835 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.040838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.041021 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.041724 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:26.041742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:26.043285 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:26.045427 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:26.045431 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:26.045445 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:26.045793 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:26.046104 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:26.046106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:26.046108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:26.046245 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:26.046759 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:26.046764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:26.047025 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 02:35:26.047239 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 02:35:26.047526 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260619 02:35:26.049814 18402 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:35:26.050112 18402 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:26.050642 18402 heartbeater.cc:511] Master 127.7.194.254:38209 requested a full tablet report, sending...
I20260619 02:35:26.052397 18210 ts_manager.cc:194] Registered new tserver with Master: 49aabad8310a4761a04d92c39fb50c99 (127.7.194.193:38177)
I20260619 02:35:26.053706 18210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:58129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:35:26.066192 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38209
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:26.180435 18407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:26.180740 18407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:26.180788 18407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:26.184991 18407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:26.185074 18407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:26.185135 18407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:35:26.190726 18407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:26.192031 18407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:26.193210 18407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:26.203516 18413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:26.203513 18412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:26.203590 18407 server_base.cc:1034] running on GCE node
W20260619 02:35:26.203516 18415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:26.204185 18407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:26.204949 18407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:26.206135 18407 hybrid_clock.cc:648] HybridClock initialized: now 1781836526206110 us; error 43 us; skew 500 ppm
Jun 19 02:35:26 dist-test-slave-9tkz krb5kdc[18166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836526, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:26.209029 18407 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:35:26.210052 18407 webserver.cc:460] Webserver started at http://127.7.194.194:45399/ using document root <none> and password file <none>
I20260619 02:35:26.210650 18407 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:26.210747 18407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:26.210991 18407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:26.213251 18407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "6d259a2cb3d248c2b3533d02f27fd717"
format_stamp: "Formatted at 2026-06-19 02:35:26 on dist-test-slave-9tkz"
server_key: "c5aa079eae2eeafe281bccc39d99099f"
server_key_iv: "024bab8a9597bebbbe2120e0bb2072d2"
server_key_version: "encryptionkey@0"
I20260619 02:35:26.213892 18407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "6d259a2cb3d248c2b3533d02f27fd717"
format_stamp: "Formatted at 2026-06-19 02:35:26 on dist-test-slave-9tkz"
server_key: "c5aa079eae2eeafe281bccc39d99099f"
server_key_iv: "024bab8a9597bebbbe2120e0bb2072d2"
server_key_version: "encryptionkey@0"
I20260619 02:35:26.217926 18407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:35:26.220530 18422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:26.221969 18407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:35:26.222118 18407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "6d259a2cb3d248c2b3533d02f27fd717"
format_stamp: "Formatted at 2026-06-19 02:35:26 on dist-test-slave-9tkz"
server_key: "c5aa079eae2eeafe281bccc39d99099f"
server_key_iv: "024bab8a9597bebbbe2120e0bb2072d2"
server_key_version: "encryptionkey@0"
I20260619 02:35:26.222245 18407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:26.241004 18407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:26.244479 18407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:26.244704 18407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:26.245347 18407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:26.246299 18407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:26.246371 18407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:26.246444 18407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:26.246493 18407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:26.255842 18407 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:39489
I20260619 02:35:26.255867 18535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:39489 every 8 connection(s)
I20260619 02:35:26.256877 18407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:35:26.263234 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 18407
I20260619 02:35:26.263360 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:35:26.263695 7947 external_mini_cluster.cc:1428] Setting key ef802db48404c0d40231e6e9b7b323b5
Jun 19 02:35:26 dist-test-slave-9tkz krb5kdc[18166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836526, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:35:26.271837 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.258804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:50611 (local address 127.7.194.254:38209)
0619 02:35:26.258935 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.258945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.259826 (+ 881us) server_negotiation.cc:407] Connection header received
0619 02:35:26.261253 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.261257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.261317 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.261456 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:26.262739 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.263484 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.264950 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.265265 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.265379 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.267985 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:26.267990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:26.267993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:26.268028 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:26.269631 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:26.270217 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:26.270221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:26.270223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:26.270323 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:26.270681 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:26.270686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:26.270688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:26.270888 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:26.270999 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.271533 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.271665 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:35:26.272750 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.259217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38209 (local address 127.7.194.194:50611)
0619 02:35:26.259667 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:26.259711 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:26.261028 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:26.261681 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:26.261689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:26.262078 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:26.262583 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.262597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.263685 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.263688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.264834 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.264844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.265427 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.265430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.265651 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.266313 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:26.266331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:26.267803 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:26.269753 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:26.269759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:26.269778 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:26.270099 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:26.270420 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:26.270424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:26.270426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:26.270581 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:26.270997 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:26.271006 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:26.271279 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 02:35:26.271482 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 02:35:26.271858 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:35:26.274173 18536 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38209
I20260619 02:35:26.274439 18536 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:26.274973 18536 heartbeater.cc:511] Master 127.7.194.254:38209 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:35:26.276182 18210 ts_manager.cc:194] Registered new tserver with Master: 6d259a2cb3d248c2b3533d02f27fd717 (127.7.194.194:39489)
I20260619 02:35:26.277043 18210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:50611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:35:26.292193 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38209
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:35:26.406231 18541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:35:26.406551 18541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:35:26.406633 18541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:35:26.410858 18541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:35:26.410940 18541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:35:26.411019 18541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:35:26.416499 18541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:35:26.417732 18541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:35:26.418861 18541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:35:26.429162 18547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:35:26.429160 18549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:26.429262 18541 server_base.cc:1034] running on GCE node
W20260619 02:35:26.429181 18546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:35:26.429940 18541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:35:26.430622 18541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:35:26.431807 18541 hybrid_clock.cc:648] HybridClock initialized: now 1781836526431768 us; error 52 us; skew 500 ppm
Jun 19 02:35:26 dist-test-slave-9tkz krb5kdc[18166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836526, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:35:26.434676 18541 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:35:26.435695 18541 webserver.cc:460] Webserver started at http://127.7.194.195:45535/ using document root <none> and password file <none>
I20260619 02:35:26.436300 18541 fs_manager.cc:359] Metadata directory not provided
I20260619 02:35:26.436401 18541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:35:26.436643 18541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:35:26.438973 18541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "c9f8af81b11b4979a21792fd89308152"
format_stamp: "Formatted at 2026-06-19 02:35:26 on dist-test-slave-9tkz"
server_key: "c81b4c0f8677d2b52e8cc8ead98c14c5"
server_key_iv: "f9ad5df8c4c2e2d28f694ff8d0f4e0d1"
server_key_version: "encryptionkey@0"
I20260619 02:35:26.439579 18541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "c9f8af81b11b4979a21792fd89308152"
format_stamp: "Formatted at 2026-06-19 02:35:26 on dist-test-slave-9tkz"
server_key: "c81b4c0f8677d2b52e8cc8ead98c14c5"
server_key_iv: "f9ad5df8c4c2e2d28f694ff8d0f4e0d1"
server_key_version: "encryptionkey@0"
I20260619 02:35:26.443567 18541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:35:26.446262 18556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:26.447628 18541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:35:26.447784 18541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "c9f8af81b11b4979a21792fd89308152"
format_stamp: "Formatted at 2026-06-19 02:35:26 on dist-test-slave-9tkz"
server_key: "c81b4c0f8677d2b52e8cc8ead98c14c5"
server_key_iv: "f9ad5df8c4c2e2d28f694ff8d0f4e0d1"
server_key_version: "encryptionkey@0"
I20260619 02:35:26.447913 18541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:35:26.480736 18541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:35:26.483940 18541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:35:26.484164 18541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:35:26.484808 18541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:35:26.485719 18541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:35:26.485795 18541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:26.485872 18541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:35:26.485924 18541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:35:26.496063 18541 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:39875
I20260619 02:35:26.496080 18669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:39875 every 8 connection(s)
I20260619 02:35:26.497059 18541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:35:26.499434 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 18541
I20260619 02:35:26.499575 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:35:26.499922 7947 external_mini_cluster.cc:1428] Setting key e2316625ac5df89f04a6e2c0f3a63eef
Jun 19 02:35:26 dist-test-slave-9tkz krb5kdc[18166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836526, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:35:26.517091 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.499024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:42251 (local address 127.7.194.254:38209)
0619 02:35:26.499166 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.499170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.500296 (+ 1126us) server_negotiation.cc:407] Connection header received
0619 02:35:26.502011 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.502014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.502071 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.502148 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:26.503400 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.503942 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.505119 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.505357 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.505479 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.508001 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:26.508005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:26.508010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:26.508042 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:26.514870 (+ 6828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:26.515466 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:26.515469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:26.515470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:26.515552 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:26.515981 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:26.515983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:26.515984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:26.516167 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:26.516241 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.516857 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.516966 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:35:26.518227 18672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.499536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38209 (local address 127.7.194.195:42251)
0619 02:35:26.500129 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:26.500175 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:26.501786 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:26.502335 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:26.502344 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:26.502732 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:35:26.503247 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.503262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.504074 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.504084 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.504988 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.504997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.505511 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.505515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.505720 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.506294 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:35:26.506310 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:35:26.507792 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:35:26.515002 (+ 7210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:26.515009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:26.515027 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:26.515363 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:26.515667 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:35:26.515669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:35:26.515671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:35:26.515892 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:35:26.516283 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:35:26.516290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:35:26.516609 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 02:35:26.516819 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 02:35:26.517126 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":144,"threads_started":1}
I20260619 02:35:26.519474 18670 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38209
I20260619 02:35:26.519717 18670 heartbeater.cc:464] Registering TS with master...
I20260619 02:35:26.520207 18670 heartbeater.cc:511] Master 127.7.194.254:38209 requested a full tablet report, sending...
I20260619 02:35:26.521274 18210 ts_manager.cc:194] Registered new tserver with Master: c9f8af81b11b4979a21792fd89308152 (127.7.194.195:39875)
I20260619 02:35:26.521961 18210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:42251
I20260619 02:35:26.526281 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:35:26.535768 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.528321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.7.194.254:38209)
0619 02:35:26.528458 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.528462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.528639 (+ 177us) server_negotiation.cc:407] Connection header received
0619 02:35:26.528742 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.528744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.528781 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.528848 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:35:26.529581 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.530103 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.530940 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.531142 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.531217 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.532785 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:35:26.532788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:35:26.532790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:35:26.532808 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:35:26.534207 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:26.534637 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:26.534639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:26.534641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:26.534714 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:35:26.534997 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:35:26.534999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:35:26.535000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:35:26.535211 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:35:26.535311 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.535533 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.535638 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:35:26.538888 18210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:35:26.542141 18210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:35:26.558538 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.551929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.7.194.194:39489)
0619 02:35:26.552654 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.552667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.553271 (+ 604us) server_negotiation.cc:407] Connection header received
0619 02:35:26.553596 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.553604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.553843 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.554009 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:26.554320 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.555396 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.557075 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.557759 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.557881 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.558043 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.558232 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.558348 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":116,"threads_started":1}
I20260619 02:35:26.558547 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.551710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38177 (local address 127.0.0.1:37054)
0619 02:35:26.552144 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:26.552200 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:26.552332 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:26.553751 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:26.553757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:26.553781 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:26.553939 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.553948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.555507 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.555511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.556926 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.556938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.558079 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.558083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.558178 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.558201 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:35:26.558253 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:35:26.558371 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":201,"thread_start_us":151,"threads_started":1}
I20260619 02:35:26.558899 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.552550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:39875 (local address 127.0.0.1:34600)
0619 02:35:26.553237 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:26.553251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:26.553365 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:26.554045 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:26.554048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:26.554060 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:26.554213 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.554217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.555381 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.555385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.556987 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.557022 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.558252 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.558255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.558613 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.558638 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:35:26.558710 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:35:26.558777 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":116,"spinlock_wait_cycles":4736,"thread_start_us":85,"threads_started":1}
I20260619 02:35:26.559096 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.552762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.7.194.193:38177)
0619 02:35:26.553052 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.553059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.553195 (+ 136us) server_negotiation.cc:407] Connection header received
0619 02:35:26.553291 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.553295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.553595 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.553752 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:26.554104 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.555357 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.557066 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.557930 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.558370 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.558791 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.558844 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.558963 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260619 02:35:26.559229 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.552700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.7.194.195:39875)
0619 02:35:26.553134 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.553144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.553347 (+ 203us) server_negotiation.cc:407] Connection header received
0619 02:35:26.553558 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.553565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.553739 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.553865 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:26.554383 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.555192 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.557146 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.557930 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.558050 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.558194 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.558996 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.559084 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":167,"threads_started":1}
I20260619 02:35:26.558528 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.551710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:39489 (local address 127.0.0.1:35770)
0619 02:35:26.553144 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:26.553159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:26.553365 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:26.553996 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:26.553999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:26.554020 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:26.554200 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.554208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.555529 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.555532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.556927 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.556938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.557893 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.557896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.558004 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.558039 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:35:26.558187 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 02:35:26.558348 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"mutex_wait_us":246,"spinlock_wait_cycles":5248,"thread_start_us":120,"threads_started":1}
I20260619 02:35:26.562104 18336 tablet_service.cc:1467] Processing CreateTablet for tablet 836094b14bf5481eb11eeb720e9e5d98 (DEFAULT_TABLE table=test-table [id=dd417d035452429cb9708fee7a290e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:26.562134 18604 tablet_service.cc:1467] Processing CreateTablet for tablet 836094b14bf5481eb11eeb720e9e5d98 (DEFAULT_TABLE table=test-table [id=dd417d035452429cb9708fee7a290e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:26.562394 18470 tablet_service.cc:1467] Processing CreateTablet for tablet 836094b14bf5481eb11eeb720e9e5d98 (DEFAULT_TABLE table=test-table [id=dd417d035452429cb9708fee7a290e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:35:26.563158 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836094b14bf5481eb11eeb720e9e5d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:26.563184 18604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836094b14bf5481eb11eeb720e9e5d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:26.563225 18470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836094b14bf5481eb11eeb720e9e5d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:35:26.570451 18689 tablet_bootstrap.cc:492] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: Bootstrap starting.
I20260619 02:35:26.571120 18688 tablet_bootstrap.cc:492] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: Bootstrap starting.
I20260619 02:35:26.569695 18687 tablet_bootstrap.cc:492] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99: Bootstrap starting.
I20260619 02:35:26.573273 18688 tablet_bootstrap.cc:654] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:26.573390 18689 tablet_bootstrap.cc:654] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:26.573491 18687 tablet_bootstrap.cc:654] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99: Neither blocks nor log segments found. Creating new log.
I20260619 02:35:26.574124 18688 log.cc:824] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:26.574311 18687 log.cc:824] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:26.574299 18689 log.cc:824] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: Log is configured to *not* fsync() on all Append() calls
I20260619 02:35:26.576244 18689 tablet_bootstrap.cc:492] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: No bootstrap required, opened a new log
I20260619 02:35:26.576267 18688 tablet_bootstrap.cc:492] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: No bootstrap required, opened a new log
I20260619 02:35:26.576267 18687 tablet_bootstrap.cc:492] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99: No bootstrap required, opened a new log
I20260619 02:35:26.576467 18688 ts_tablet_manager.cc:1397] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 02:35:26.576472 18689 ts_tablet_manager.cc:1397] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:35:26.576694 18687 ts_tablet_manager.cc:1397] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 02:35:26.579641 18689 raft_consensus.cc:348] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } }
I20260619 02:35:26.579911 18689 raft_consensus.cc:374] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:26.579974 18689 raft_consensus.cc:729] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d259a2cb3d248c2b3533d02f27fd717, State: Initialized, Role: FOLLOWER
I20260619 02:35:26.580453 18689 consensus_queue.cc:260] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } }
I20260619 02:35:26.580652 18688 raft_consensus.cc:348] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } }
I20260619 02:35:26.580937 18688 raft_consensus.cc:374] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:26.580996 18688 raft_consensus.cc:729] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f8af81b11b4979a21792fd89308152, State: Initialized, Role: FOLLOWER
I20260619 02:35:26.581079 18687 raft_consensus.cc:348] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } }
I20260619 02:35:26.581395 18687 raft_consensus.cc:374] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:35:26.581446 18687 raft_consensus.cc:729] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49aabad8310a4761a04d92c39fb50c99, State: Initialized, Role: FOLLOWER
I20260619 02:35:26.581653 18689 ts_tablet_manager.cc:1428] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:35:26.581672 18688 consensus_queue.cc:260] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } }
I20260619 02:35:26.581895 18687 consensus_queue.cc:260] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } }
I20260619 02:35:26.582494 18670 heartbeater.cc:503] Master 127.7.194.254:38209 was elected leader, sending a full tablet report...
I20260619 02:35:26.582882 18687 ts_tablet_manager.cc:1428] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:35:26.582948 18688 ts_tablet_manager.cc:1428] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: Time spent starting tablet: real 0.006s user 0.000s sys 0.008s
I20260619 02:35:26.583222 18402 heartbeater.cc:503] Master 127.7.194.254:38209 was elected leader, sending a full tablet report...
I20260619 02:35:26.585783 18536 heartbeater.cc:503] Master 127.7.194.254:38209 was elected leader, sending a full tablet report...
I20260619 02:35:26.663180 18694 raft_consensus.cc:482] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:35:26.663461 18694 raft_consensus.cc:504] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } }
I20260619 02:35:26.664716 18694 leader_election.cc:288] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d259a2cb3d248c2b3533d02f27fd717 (127.7.194.194:39489), 49aabad8310a4761a04d92c39fb50c99 (127.7.194.193:38177)
I20260619 02:35:26.669373 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.665186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:58275 (local address 127.7.194.193:38177)
0619 02:35:26.665314 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.665318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.665341 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:35:26.665452 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.665456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.665518 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.665667 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:26.665968 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.666773 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.668291 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.668763 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.668897 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.668953 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.669152 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.669223 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:35:26.669373 18672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.665049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38177 (local address 127.7.194.195:58275)
0619 02:35:26.665196 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:26.665215 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:26.665318 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:26.665690 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:26.665693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:26.665715 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:26.665839 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.665847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.666882 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.666885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.668151 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.668163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.668897 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.668901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.669019 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.669037 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:35:26.669122 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:35:26.669205 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":22}
I20260619 02:35:26.669410 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.665153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:38665 (local address 127.7.194.194:39489)
0619 02:35:26.665290 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.665294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.665938 (+ 644us) server_negotiation.cc:407] Connection header received
0619 02:35:26.666014 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.666017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.666069 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.666152 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:35:26.666383 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.667045 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.668434 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.668840 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.668979 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.669032 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.669213 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.669287 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:35:26.669422 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.665068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:39489 (local address 127.7.194.195:38665)
0619 02:35:26.665824 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:26.665837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:35:26.665934 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:35:26.666160 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:35:26.666161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:35:26.666176 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:35:26.666264 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.666268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.667141 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.667144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.668326 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:35:26.668332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.668975 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.668979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:35:26.669087 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.669110 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:35:26.669196 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:35:26.669266 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":304,"thread_start_us":203,"threads_started":1}
I20260619 02:35:26.670077 18356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836094b14bf5481eb11eeb720e9e5d98" candidate_uuid: "c9f8af81b11b4979a21792fd89308152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49aabad8310a4761a04d92c39fb50c99" is_pre_election: true
I20260619 02:35:26.670079 18490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836094b14bf5481eb11eeb720e9e5d98" candidate_uuid: "c9f8af81b11b4979a21792fd89308152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d259a2cb3d248c2b3533d02f27fd717" is_pre_election: true
I20260619 02:35:26.670392 18356 raft_consensus.cc:2454] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f8af81b11b4979a21792fd89308152 in term 0.
I20260619 02:35:26.670406 18490 raft_consensus.cc:2454] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f8af81b11b4979a21792fd89308152 in term 0.
I20260619 02:35:26.670894 18558 leader_election.cc:302] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49aabad8310a4761a04d92c39fb50c99, c9f8af81b11b4979a21792fd89308152; no voters:
I20260619 02:35:26.671218 18694 raft_consensus.cc:2781] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:35:26.671303 18694 raft_consensus.cc:482] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:35:26.671347 18694 raft_consensus.cc:3037] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:26.672364 18694 raft_consensus.cc:504] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } }
I20260619 02:35:26.672703 18694 leader_election.cc:288] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [CANDIDATE]: Term 1 election: Requested vote from peers 6d259a2cb3d248c2b3533d02f27fd717 (127.7.194.194:39489), 49aabad8310a4761a04d92c39fb50c99 (127.7.194.193:38177)
I20260619 02:35:26.673060 18356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836094b14bf5481eb11eeb720e9e5d98" candidate_uuid: "c9f8af81b11b4979a21792fd89308152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49aabad8310a4761a04d92c39fb50c99"
I20260619 02:35:26.673059 18490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836094b14bf5481eb11eeb720e9e5d98" candidate_uuid: "c9f8af81b11b4979a21792fd89308152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d259a2cb3d248c2b3533d02f27fd717"
I20260619 02:35:26.673187 18490 raft_consensus.cc:3037] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:26.673187 18356 raft_consensus.cc:3037] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:35:26.674532 18356 raft_consensus.cc:2454] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f8af81b11b4979a21792fd89308152 in term 1.
I20260619 02:35:26.674532 18490 raft_consensus.cc:2454] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f8af81b11b4979a21792fd89308152 in term 1.
I20260619 02:35:26.674826 18558 leader_election.cc:302] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49aabad8310a4761a04d92c39fb50c99, c9f8af81b11b4979a21792fd89308152; no voters:
I20260619 02:35:26.675001 18694 raft_consensus.cc:2781] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:35:26.675235 18694 raft_consensus.cc:686] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 1 LEADER]: Becoming Leader. State: Replica: c9f8af81b11b4979a21792fd89308152, State: Running, Role: LEADER
I20260619 02:35:26.675619 18694 consensus_queue.cc:237] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } }
I20260619 02:35:26.679019 18206 catalog_manager.cc:5495] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f8af81b11b4979a21792fd89308152 (127.7.194.195). New cstate: current_term: 1 leader_uuid: "c9f8af81b11b4979a21792fd89308152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f8af81b11b4979a21792fd89308152" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39875 } health_report { overall_health: HEALTHY } } }
I20260619 02:35:26.716429 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.712793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.7.194.195:39875)
0619 02:35:26.712932 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:35:26.712936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:35:26.712949 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:35:26.713025 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:35:26.713027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:35:26.713082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:35:26.713179 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:35:26.713394 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.714055 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.715020 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:35:26.715220 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:35:26.715292 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:35:26.715552 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:35:26.715993 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:35:26.716090 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:35:26.716231 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:35:26.716284 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:35:26.723292 18490 raft_consensus.cc:1261] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [term 1 FOLLOWER]: Refusing update from remote peer c9f8af81b11b4979a21792fd89308152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:26.723315 18356 raft_consensus.cc:1261] T 836094b14bf5481eb11eeb720e9e5d98 P 49aabad8310a4761a04d92c39fb50c99 [term 1 FOLLOWER]: Refusing update from remote peer c9f8af81b11b4979a21792fd89308152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:35:26.724144 18697 consensus_queue.cc:1035] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49aabad8310a4761a04d92c39fb50c99" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:26.724355 18694 consensus_queue.cc:1035] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d259a2cb3d248c2b3533d02f27fd717" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 39489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:35:26.733829 18700 mvcc.cc:205] Tried to move back new op lower bound from 7298402413450203136 to 7298402413266350080. Current Snapshot: MvccSnapshot[applied={T|T < 7298402413450203136}]
I20260619 02:35:26.735560 18702 mvcc.cc:205] Tried to move back new op lower bound from 7298402413450203136 to 7298402413266350080. Current Snapshot: MvccSnapshot[applied={T|T < 7298402413450203136}]
I20260619 02:35:26.744946 18206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:26.745151 18206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:35:26.747752 18206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fb60e09afe343888d4266171824e31f: Sending DeleteTablet for 3 replicas of tablet 836094b14bf5481eb11eeb720e9e5d98
I20260619 02:35:26.748701 18470 tablet_service.cc:1514] Processing DeleteTablet for tablet 836094b14bf5481eb11eeb720e9e5d98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:26 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:35770
I20260619 02:35:26.748979 18336 tablet_service.cc:1514] Processing DeleteTablet for tablet 836094b14bf5481eb11eeb720e9e5d98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:26 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:37054
I20260619 02:35:26.748998 18604 tablet_service.cc:1514] Processing DeleteTablet for tablet 836094b14bf5481eb11eeb720e9e5d98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:35:26 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:34600
I20260619 02:35:26.749262 18709 tablet_replica.cc:330] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: stopping tablet replica
I20260619 02:35:26.749464 18711 tablet_replica.cc:330] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: stopping tablet replica
I20260619 02:35:26.749632 18709 raft_consensus.cc:2229] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:35:26.749720 18711 raft_consensus.cc:2229] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:35:26.749974 18709 raft_consensus.cc:2258] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:26.750088 18711 raft_consensus.cc:2258] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:35:26.750125 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 18273
I20260619 02:35:26.752050 18709 ts_tablet_manager.cc:1905] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:26.752058 18711 ts_tablet_manager.cc:1905] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:35:26.754860 18711 ts_tablet_manager.cc:1918] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:35:26.754860 18709 ts_tablet_manager.cc:1918] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:35:26.754954 18709 log.cc:1196] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/836094b14bf5481eb11eeb720e9e5d98
I20260619 02:35:26.754954 18711 log.cc:1196] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/836094b14bf5481eb11eeb720e9e5d98
I20260619 02:35:26.755270 18709 ts_tablet_manager.cc:1939] T 836094b14bf5481eb11eeb720e9e5d98 P 6d259a2cb3d248c2b3533d02f27fd717: Deleting consensus metadata
I20260619 02:35:26.755270 18711 ts_tablet_manager.cc:1939] T 836094b14bf5481eb11eeb720e9e5d98 P c9f8af81b11b4979a21792fd89308152: Deleting consensus metadata
I20260619 02:35:26.757417 18198 catalog_manager.cc:4841] TS 6d259a2cb3d248c2b3533d02f27fd717 (127.7.194.194:39489): tablet 836094b14bf5481eb11eeb720e9e5d98 (table test-table [id=dd417d035452429cb9708fee7a290e0c]) successfully deleted
I20260619 02:35:26.757417 18196 catalog_manager.cc:4841] TS c9f8af81b11b4979a21792fd89308152 (127.7.194.195:39875): tablet 836094b14bf5481eb11eeb720e9e5d98 (table test-table [id=dd417d035452429cb9708fee7a290e0c]) successfully deleted
W20260619 02:35:26.759265 18196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:38177 (error 108)
I20260619 02:35:26.759493 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 18407
I20260619 02:35:26.760921 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:35:26.760498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38177 (local address 127.0.0.1:37066)
0619 02:35:26.760694 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 02:35:26.760769 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:38177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260619 02:35:26.761130 18196 catalog_manager.cc:4584] TS 49aabad8310a4761a04d92c39fb50c99 (127.7.194.193:38177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 836094b14bf5481eb11eeb720e9e5d98: Network error: Client connection negotiation failed: client connection to 127.7.194.193:38177: connect: Connection refused (error 111)
I20260619 02:35:26.766052 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 18541
I20260619 02:35:26.771982 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 18179
2026-06-19T02:35:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2404 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:35:26 dist-test-slave-9tkz krb5kdc[18715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:35:26 dist-test-slave-9tkz krb5kdc[18715](info): set up 2 sockets
Jun 19 02:35:26 dist-test-slave-9tkz krb5kdc[18715](info): commencing operation
krb5kdc: starting...
W20260619 02:35:27.818886 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:35:27 dist-test-slave-9tkz krb5kdc[18715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:35:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:35:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/rangeradmin_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/rangeradmin_127.7.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/rangerlookup_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/rangerlookup_127.7.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/HTTP_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/HTTP_127.7.194.212@KRBTEST.COM.keytab.
I20260619 02:35:27.992836 7947 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task73xKWu/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/postgres -l logfile start
2026-06-19 02:35:30.454 UTC [18745] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:35:30.454 UTC [18745] LOG: listening on IPv4 address "127.7.194.212", port 34851
2026-06-19 02:35:30.457 UTC [18745] LOG: listening on Unix socket "/tmp/.s.PGSQL.34851"
2026-06-19 02:35:30.484 UTC [18748] LOG: database system was shut down at 2026-06-19 02:35:29 UTC
2026-06-19 02:35:30.488 UTC [18745] LOG: database system is ready to accept connections
I20260619 02:35:31.434628 7947 mini_postgres.cc:96] Postgres bound to 34851
2026-06-19 02:35:31.441 UTC [18756] FATAL: database "slave" does not exist
127.7.194.212:34851 - accepting connections
I20260619 02:35:31.442001 7947 mini_ranger.cc:161] Starting Ranger...
I20260619 02:35:31.457852 7947 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:35:31.934806 7947 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:35:31.934906 7947 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-admin
2026-06-19 02:35:32,213 [I] DB FLAVOR :POSTGRES
2026-06-19 02:35:32,213 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:35:32,213 [I] Checking connection..
2026-06-19 02:35:32,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:35:32,462 [I] Checking connection passed.
2026-06-19 02:35:32,463 [I] --------- Verifying version history table ---------
2026-06-19 02:35:32,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:35:32,716 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:35:32,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:35:32,985 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:35:32,985 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:35:32,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:35:33.244 UTC [18820] WARNING: there is no transaction in progress
2026-06-19 02:35:33,251 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:35:33,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:35:33,491 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:35:33,492 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:35:33,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:35:33,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:35:34,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9tkz.c.gcp-upstream.internal','N') ;"
2026-06-19 02:35:34,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:35:34,494 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:35:34,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:35:34,744 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:35:34,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:35:35,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:35:35,268 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:35:35,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:35:36.304 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.321 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.337 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.353 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.370 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.390 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.455 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.462 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.476 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.487 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.498 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.512 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.521 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.527 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.534 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:36.540 UTC [18928] WARNING: there is no transaction in progress
2026-06-19 02:35:37,198 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:35:37,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:35:37,456 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:35:37,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:35:37,725 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:35:37,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:35:37,964 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:35:37,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:35:38,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:35:38,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9tkz.c.gcp-upstream.internal';"
2026-06-19 02:35:38,704 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:35:38,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:35:38,958 [I] DB_PATCHES have already been applied
I20260619 02:35:38.964339 7947 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-admin:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task73xKWu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:35:38.964416 7947 mini_ranger.cc:193] Using host: 127.7.194.212
I20260619 02:35:38.968120 7947 mini_ranger.cc:227] Ranger admin URL: http://127.7.194.212:51695
Jun 19, 2026 2:35:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:35:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:35:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:35:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:35:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:35:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:35:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:35:40.117347 7947 mini_ranger.cc:160] Time spent starting Ranger: real 8.675s user 0.000s sys 0.011s
Jun 19, 2026 2:36:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:36:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:36:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:36:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:36:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:36:02.827297 7947 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/rangerkms_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/rangerkms_127.7.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.194.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.194.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.194.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/HTTP_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.194.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/HTTP_127.7.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:36:02 dist-test-slave-9tkz krb5kdc[18715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836562, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:36:02.925431 7947 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:36:02.936506 7947 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:36:03.833264 7947 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:36:03.833344 7947 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-kms
I20260619 02:36:03.833434 7947 mini_ranger_kms.cc:227] Using postgres at 127.7.194.212:34851
2026-06-19 02:36:04,026 [I] DB FLAVOR :POSTGRES
2026-06-19 02:36:04,026 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:36:04,026 [I] Checking connection
2026-06-19 02:36:04,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:36:04,291 [I] connection success
2026-06-19 02:36:04,291 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:36:04,291 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:36:04,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:36:04,529 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:36:04,530 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:36:04,530 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:36:04,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:34851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:36:04,811 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:36:05.316361 7947 mini_ranger_kms.cc:310] Created kms service
I20260619 02:36:05.507755 7947 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:36:05.558120 7947 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:36:05 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836562, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:36:06.037449 7947 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:36:06.037819 7947 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-kms:/tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:36:06.037856 7947 mini_ranger_kms.cc:240] Using host: 127.7.194.212
I20260619 02:36:06.041831 7947 mini_ranger_kms.cc:276] Ranger KMS PID: 19109
I20260619 02:36:06.041994 7947 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.194.212:50829
Jun 19, 2026 2:36:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:36:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:36:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:36:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:36:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:36:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 02:36:08.344498 7947 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.419s user 0.014s sys 0.162s
I20260619 02:36:08.344655 7947 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:36:08.344764 7947 mini_ranger_kms.cc:401] 127.7.194.212:50829/kms/v1/keys
Jun 19 02:36:08 dist-test-slave-9tkz krb5kdc[18715](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781836568, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.194.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 02:36:08 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836568, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.194.212@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
Jun 19 02:36:09 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836568, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.194.212@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
Jun 19 02:36:11 dist-test-slave-9tkz krb5kdc[18715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:36:11.573127 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:37817
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:37817
--ranger_config_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:50829/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:36:11.693671 19159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:36:11.693958 19159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:36:11.693998 19159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:36:11.698192 19159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:36:11.698271 19159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:36:11.698290 19159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:36:11.698308 19159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:36:11.698323 19159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:36:11.704150 19159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.194.212:50829/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:37817
--ranger_config_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:37817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:36:11.705351 19159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:36:11.706548 19159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:36:11.716861 19165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:36:11.716967 19167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:36:11.717028 19159 server_base.cc:1034] running on GCE node
W20260619 02:36:11.716866 19164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:36:11.717926 19159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:36:11.719545 19159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:36:11.720754 19159 hybrid_clock.cc:648] HybridClock initialized: now 1781836571720721 us; error 55 us; skew 500 ppm
Jun 19 02:36:11 dist-test-slave-9tkz krb5kdc[18715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:36:11.729137 19159 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:36:11.730363 19159 webserver.cc:460] Webserver started at http://127.7.194.254:34183/ using document root <none> and password file <none>
I20260619 02:36:11.730984 19159 fs_manager.cc:359] Metadata directory not provided
I20260619 02:36:11.731043 19159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:36:11.731292 19159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:36:11 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:36:11.796883 19159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "6378f373ddf7473a9eea370277c6a4cf"
format_stamp: "Formatted at 2026-06-19 02:36:11 on dist-test-slave-9tkz"
server_key: "c2f0cf6f32fc4edc662460559543eaee"
server_key_iv: "fcdf03a0cd2a8244d8ec8c5bd1d6af59"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:11.797602 19159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "6378f373ddf7473a9eea370277c6a4cf"
format_stamp: "Formatted at 2026-06-19 02:36:11 on dist-test-slave-9tkz"
server_key: "c2f0cf6f32fc4edc662460559543eaee"
server_key_iv: "fcdf03a0cd2a8244d8ec8c5bd1d6af59"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:11.802526 19159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:36:11.821910 19174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:11.823450 19159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:36:11.823607 19159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "6378f373ddf7473a9eea370277c6a4cf"
format_stamp: "Formatted at 2026-06-19 02:36:11 on dist-test-slave-9tkz"
server_key: "c2f0cf6f32fc4edc662460559543eaee"
server_key_iv: "fcdf03a0cd2a8244d8ec8c5bd1d6af59"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:11.823762 19159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:36:11.835994 19159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:36:11.839167 19159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:36:11.839386 19159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:36:11.848235 19226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:37817 every 8 connection(s)
I20260619 02:36:11.848237 19159 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:37817
I20260619 02:36:11.849352 19159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:36:11.851909 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 19159
I20260619 02:36:11.852046 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:36:11.852344 7947 external_mini_cluster.cc:1428] Setting key e8dae54518d664f64c0e4a7fbf69c0c4
I20260619 02:36:11.853080 19227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:36:11.860457 19227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf: Bootstrap starting.
Jun 19 02:36:11 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:36:11.863302 19227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf: Neither blocks nor log segments found. Creating new log.
I20260619 02:36:11.864699 19227 log.cc:824] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf: Log is configured to *not* fsync() on all Append() calls
I20260619 02:36:11.866995 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:11.853918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.7.194.254:37817)
0619 02:36:11.854334 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:11.854352 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:11.854401 (+ 49us) server_negotiation.cc:407] Connection header received
0619 02:36:11.855058 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:11.855075 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:11.855411 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:11.855762 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:36:11.856550 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:11.857614 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:11.858496 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:11.858985 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:11.859160 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:11.861813 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:36:11.861820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:36:11.861845 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:36:11.861892 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:36:11.863744 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:11.864228 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:11.864235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:11.864248 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:11.864378 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:11.864680 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:11.864689 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:11.864691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:11.865231 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:36:11.865481 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:11.865760 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 02:36:11.866061 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":148,"threads_started":1}
I20260619 02:36:11.868048 19227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf: No bootstrap required, opened a new log
I20260619 02:36:11.871325 19227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6378f373ddf7473a9eea370277c6a4cf" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37817 } }
I20260619 02:36:11.871599 19227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:36:11.871680 19227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6378f373ddf7473a9eea370277c6a4cf, State: Initialized, Role: FOLLOWER
I20260619 02:36:11.872221 19227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6378f373ddf7473a9eea370277c6a4cf" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37817 } }
I20260619 02:36:11.872404 19227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:36:11.872467 19227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:36:11.872576 19227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:36:11.873693 19227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6378f373ddf7473a9eea370277c6a4cf" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37817 } }
I20260619 02:36:11.874047 19227 leader_election.cc:302] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6378f373ddf7473a9eea370277c6a4cf; no voters:
I20260619 02:36:11.874434 19227 leader_election.cc:288] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:36:11.874550 19232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:36:11.874776 19232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [term 1 LEADER]: Becoming Leader. State: Replica: 6378f373ddf7473a9eea370277c6a4cf, State: Running, Role: LEADER
I20260619 02:36:11.875095 19232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6378f373ddf7473a9eea370277c6a4cf" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37817 } }
I20260619 02:36:11.875909 19227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:36:11.876485 19232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6378f373ddf7473a9eea370277c6a4cf. Latest consensus state: current_term: 1 leader_uuid: "6378f373ddf7473a9eea370277c6a4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6378f373ddf7473a9eea370277c6a4cf" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37817 } } }
I20260619 02:36:11.876601 19232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [sys.catalog]: This master's current role is: LEADER
I20260619 02:36:11.877090 19234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6378f373ddf7473a9eea370277c6a4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6378f373ddf7473a9eea370277c6a4cf" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37817 } } }
I20260619 02:36:11.877252 19234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf [sys.catalog]: This master's current role is: LEADER
I20260619 02:36:11.878381 19227 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:36:11.878866 19237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:36:11.885325 19237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:36:11.893707 19237 catalog_manager.cc:1347] Generated new cluster ID: ae9d7389b88d45508b80c57159b76d79
I20260619 02:36:11.893793 19237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:36:11.912843 19237 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:36:11.914311 19237 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:36:11.921329 19237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf: Generated new TSK 0
I20260619 02:36:11.922254 19237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:36:12.621230 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:50829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:37817
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:36:12.763409 19272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:36:12.763846 19272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:36:12.763921 19272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:36:12.770201 19272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:36:12.770340 19272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:36:12.770452 19272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:36:12.778597 19272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.194.212:50829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:37817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:36:12.779976 19272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:36:12.781481 19272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:36:12.792245 19280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:36:12.792356 19272 server_base.cc:1034] running on GCE node
W20260619 02:36:12.792446 19282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:36:12.792768 19279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:36:12.793216 19272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:36:12.794013 19272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:36:12.795192 19272 hybrid_clock.cc:648] HybridClock initialized: now 1781836572795164 us; error 52 us; skew 500 ppm
Jun 19 02:36:12 dist-test-slave-9tkz krb5kdc[18715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836572, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:36:12.799661 19272 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:36:12.800606 19272 webserver.cc:460] Webserver started at http://127.7.194.193:45931/ using document root <none> and password file <none>
I20260619 02:36:12.801204 19272 fs_manager.cc:359] Metadata directory not provided
I20260619 02:36:12.801266 19272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:36:12.801486 19272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:36:12 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836572, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:36:12.830976 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "a66092dfd11549afb9e109e2a1182393"
format_stamp: "Formatted at 2026-06-19 02:36:12 on dist-test-slave-9tkz"
server_key: "392b7a49ae39140e70b9006a39649939"
server_key_iv: "73b86722b9d2e056a095a9e11fd4e39e"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:12.831907 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "a66092dfd11549afb9e109e2a1182393"
format_stamp: "Formatted at 2026-06-19 02:36:12 on dist-test-slave-9tkz"
server_key: "392b7a49ae39140e70b9006a39649939"
server_key_iv: "73b86722b9d2e056a095a9e11fd4e39e"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:12.838238 19272 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 02:36:12.873972 19289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:12.878315 19272 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.001s
I20260619 02:36:12.878500 19272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "a66092dfd11549afb9e109e2a1182393"
format_stamp: "Formatted at 2026-06-19 02:36:12 on dist-test-slave-9tkz"
server_key: "392b7a49ae39140e70b9006a39649939"
server_key_iv: "73b86722b9d2e056a095a9e11fd4e39e"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:12.878682 19272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:36:12.896814 19272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:36:12.900815 19272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:36:12.901129 19272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:36:12.901988 19272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:36:12.903118 19272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:36:12.903225 19272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:12.903350 19272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:36:12.903429 19272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:12.917383 19272 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:46703
I20260619 02:36:12.918730 19272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:36:12.920336 19404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:46703 every 8 connection(s)
I20260619 02:36:12.921970 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 19272
I20260619 02:36:12.922094 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:36:12.922432 7947 external_mini_cluster.cc:1428] Setting key 1301506384133e245a932a40134eb313
Jun 19 02:36:12 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836572, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:36:12.944131 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:12.923687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:37817 (local address 127.7.194.193:46505)
0619 02:36:12.925133 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:12.925192 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:12.926264 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:12.930405 (+ 4141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:12.930419 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:12.930828 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:36:12.931601 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:12.931625 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:12.932805 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:12.932809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:12.934283 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:12.934294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:12.935093 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:12.935097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:12.935309 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:12.936031 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:36:12.936054 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:36:12.937914 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:36:12.940302 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:36:12.940309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:36:12.940330 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:36:12.940749 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:36:12.941106 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:36:12.941109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:36:12.941112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:36:12.941313 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:36:12.941877 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:36:12.941895 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:36:12.942260 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 02:36:12.942488 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 02:36:12.942900 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"thread_start_us":175,"threads_started":1}
I20260619 02:36:12.944960 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:12.924847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:46505 (local address 127.7.194.254:37817)
0619 02:36:12.929777 (+ 4930us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:12.929782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:12.929803 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:36:12.929853 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:12.929856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:12.929922 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:12.930025 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:36:12.931807 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:12.932639 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:12.934437 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:12.934784 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:12.934895 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:12.938150 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:36:12.938155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:36:12.938157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:36:12.938188 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:36:12.940154 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:12.940879 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:12.940895 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:12.940897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:12.940995 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:12.941437 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:12.941442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:12.941443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:12.941717 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:36:12.941822 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:12.944631 (+ 2809us) server_negotiation.cc:299] Negotiation successful
0619 02:36:12.944773 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4831,"thread_start_us":94,"threads_started":1}
I20260619 02:36:12.946875 19405 heartbeater.cc:347] Connected to a master server at 127.7.194.254:37817
I20260619 02:36:12.947151 19405 heartbeater.cc:464] Registering TS with master...
I20260619 02:36:12.947750 19405 heartbeater.cc:511] Master 127.7.194.254:37817 requested a full tablet report, sending...
I20260619 02:36:12.949657 19191 ts_manager.cc:194] Registered new tserver with Master: a66092dfd11549afb9e109e2a1182393 (127.7.194.193:46703)
I20260619 02:36:12.951198 19191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:46505
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:36:12.984766 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:50829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:37817
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:36:12 dist-test-slave-9tkz krb5kdc[18715](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781836572, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 02:36:13.159549 19411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:36:13.160044 19411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:36:13.160154 19411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:36:13.166272 19411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:36:13.166363 19411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:36:13.166452 19411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:36:13.174423 19411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.194.212:50829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:37817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:36:13.176374 19411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:36:13.178165 19411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:36:13.192651 19418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:36:13.193173 19417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:36:13.193418 19411 server_base.cc:1034] running on GCE node
W20260619 02:36:13.193722 19420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:36:13.194270 19411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:36:13.195123 19411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:36:13.196350 19411 hybrid_clock.cc:648] HybridClock initialized: now 1781836573196281 us; error 91 us; skew 500 ppm
Jun 19 02:36:13 dist-test-slave-9tkz krb5kdc[18715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:36:13.200248 19411 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:36:13.201378 19411 webserver.cc:460] Webserver started at http://127.7.194.194:39653/ using document root <none> and password file <none>
I20260619 02:36:13.202121 19411 fs_manager.cc:359] Metadata directory not provided
I20260619 02:36:13.202198 19411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:36:13.202419 19411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:36:13 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:36:13.223990 19411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "76bd5e516d9a4b88a10cdf5497993540"
format_stamp: "Formatted at 2026-06-19 02:36:13 on dist-test-slave-9tkz"
server_key: "f26f4e21d4045a86df533ffe4ce13516"
server_key_iv: "5e5df2bebfabda60cd52894460cd3256"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:13.225937 19411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "76bd5e516d9a4b88a10cdf5497993540"
format_stamp: "Formatted at 2026-06-19 02:36:13 on dist-test-slave-9tkz"
server_key: "f26f4e21d4045a86df533ffe4ce13516"
server_key_iv: "5e5df2bebfabda60cd52894460cd3256"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:13.232141 19411 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 02:36:13.253890 19427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:13.255754 19411 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260619 02:36:13.255892 19411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "76bd5e516d9a4b88a10cdf5497993540"
format_stamp: "Formatted at 2026-06-19 02:36:13 on dist-test-slave-9tkz"
server_key: "f26f4e21d4045a86df533ffe4ce13516"
server_key_iv: "5e5df2bebfabda60cd52894460cd3256"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:13.256026 19411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:36:13.274452 19411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:36:13.278889 19411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:36:13.279187 19411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:36:13.279964 19411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:36:13.281180 19411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:36:13.281306 19411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:13.281414 19411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:36:13.281483 19411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:13.294401 19411 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:38481
I20260619 02:36:13.294456 19540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:38481 every 8 connection(s)
I20260619 02:36:13.295912 19411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:36:13.300897 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 19411
I20260619 02:36:13.301054 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:36:13.301414 7947 external_mini_cluster.cc:1428] Setting key d845640bfe2e70acf57915d466cb1f3c
Jun 19 02:36:13 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:36:13.322844 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.301925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:56201 (local address 127.7.194.254:37817)
0619 02:36:13.302116 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:13.302121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:13.306270 (+ 4149us) server_negotiation.cc:407] Connection header received
0619 02:36:13.309124 (+ 2854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:13.309128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:13.309192 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:13.309256 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:36:13.311180 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.312013 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.313877 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.314232 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.314316 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.317809 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:36:13.317814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:36:13.317817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:36:13.317865 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:36:13.320013 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:13.320738 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:13.320743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:13.320745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:13.320855 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:13.321307 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:13.321310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:13.321312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:13.321582 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:36:13.321689 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:13.322466 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 02:36:13.322616 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 02:36:13.324844 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.300568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:37817 (local address 127.7.194.194:56201)
0619 02:36:13.306076 (+ 5508us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:13.306151 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:13.308802 (+ 2651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:13.309807 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:13.309819 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:13.310228 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:36:13.310975 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.310996 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.312209 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.312213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.313703 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.313716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.314557 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.314561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.314779 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.315579 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:36:13.315604 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:36:13.317526 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:36:13.320189 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:36:13.320196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:36:13.320217 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:36:13.320603 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:36:13.320979 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:36:13.320983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:36:13.320985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:36:13.321187 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:36:13.321760 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:36:13.321767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:36:13.322156 (+ 389us) client_negotiation.cc:769] Sending connection context
0619 02:36:13.323116 (+ 960us) client_negotiation.cc:240] Negotiation successful
0619 02:36:13.323499 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5244,"thread_start_us":193,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:36:13.326349 19541 heartbeater.cc:347] Connected to a master server at 127.7.194.254:37817
I20260619 02:36:13.326622 19541 heartbeater.cc:464] Registering TS with master...
I20260619 02:36:13.327288 19541 heartbeater.cc:511] Master 127.7.194.254:37817 requested a full tablet report, sending...
I20260619 02:36:13.328575 19191 ts_manager.cc:194] Registered new tserver with Master: 76bd5e516d9a4b88a10cdf5497993540 (127.7.194.194:38481)
I20260619 02:36:13.329384 19191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:56201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:36:13.352526 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:50829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:37817
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:36:13.519904 19546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:36:13.520200 19546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:36:13.520282 19546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:36:13.524787 19546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:36:13.524866 19546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:36:13.524981 19546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:36:13.531262 19546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.194.212:50829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:37817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:36:13.532493 19546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:36:13.533699 19546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:36:13.543841 19552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:36:13.543928 19554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:36:13.544170 19551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:36:13.545202 19546 server_base.cc:1034] running on GCE node
I20260619 02:36:13.545750 19546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:36:13.546494 19546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:36:13.549814 19546 hybrid_clock.cc:648] HybridClock initialized: now 1781836573549545 us; error 273 us; skew 500 ppm
Jun 19 02:36:13 dist-test-slave-9tkz krb5kdc[18715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:36:13.554018 19546 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:36:13.555243 19546 webserver.cc:460] Webserver started at http://127.7.194.195:38953/ using document root <none> and password file <none>
I20260619 02:36:13.555917 19546 fs_manager.cc:359] Metadata directory not provided
I20260619 02:36:13.556016 19546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:36:13.556246 19546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:36:13 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:36:13.582080 19546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "67f5adc2ade44d0abf3798d124c43bd6"
format_stamp: "Formatted at 2026-06-19 02:36:13 on dist-test-slave-9tkz"
server_key: "0f3db5920ca746bed906a7aaccb95dc5"
server_key_iv: "20ece2dd0bb91af0b47c9585666ebfc7"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:13.583245 19546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "67f5adc2ade44d0abf3798d124c43bd6"
format_stamp: "Formatted at 2026-06-19 02:36:13 on dist-test-slave-9tkz"
server_key: "0f3db5920ca746bed906a7aaccb95dc5"
server_key_iv: "20ece2dd0bb91af0b47c9585666ebfc7"
server_key_version: "kuduclusterkey@0"
Jun 19 02:36:13 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836572, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:36:13.600039 19546 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.001s sys 0.003s
I20260619 02:36:13.619454 19561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:13.621377 19546 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 02:36:13.621623 19546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "67f5adc2ade44d0abf3798d124c43bd6"
format_stamp: "Formatted at 2026-06-19 02:36:13 on dist-test-slave-9tkz"
server_key: "0f3db5920ca746bed906a7aaccb95dc5"
server_key_iv: "20ece2dd0bb91af0b47c9585666ebfc7"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:13.621855 19546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:36:13.651222 19546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:36:13.655277 19546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:36:13.655536 19546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:36:13.656304 19546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:36:13.657423 19546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:36:13.657527 19546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:13.657649 19546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:36:13.657712 19546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:13.671574 19546 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:45003
I20260619 02:36:13.673233 19546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:36:13.675451 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 19546
I20260619 02:36:13.675570 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:36:13.675899 7947 external_mini_cluster.cc:1428] Setting key 25179fb8268d6c94f32c8d80e69377ef
I20260619 02:36:13.674896 19674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:45003 every 8 connection(s)
Jun 19 02:36:13 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:36:13.695088 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.679414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:39745 (local address 127.7.194.254:37817)
0619 02:36:13.679522 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:13.679527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:13.679542 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:36:13.680255 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:13.680258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:13.680311 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:13.680364 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:36:13.682720 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.683565 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.685364 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.685716 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.685793 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.689089 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:36:13.689093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:36:13.689096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:36:13.689128 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:36:13.691635 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:13.693080 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:13.693084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:13.693086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:13.693177 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:13.693665 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:13.693668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:13.693669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:13.693914 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:36:13.693984 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:13.694772 (+ 788us) server_negotiation.cc:299] Negotiation successful
0619 02:36:13.694917 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:36:13.697137 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.678297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:37817 (local address 127.7.194.195:39745)
0619 02:36:13.678901 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:13.678964 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:13.679998 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:13.680849 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:13.680860 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:13.681258 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:36:13.682533 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.682556 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.683731 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.683735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.685227 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.685237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.686020 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.686023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.686240 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.686980 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:36:13.687001 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:36:13.688864 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:36:13.692514 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:36:13.692521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:36:13.692544 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:36:13.692957 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:36:13.693282 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:36:13.693285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:36:13.693287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:36:13.693558 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:36:13.694064 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:36:13.694071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:36:13.694429 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 02:36:13.695570 (+ 1141us) client_negotiation.cc:240] Negotiation successful
0619 02:36:13.695970 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":141,"threads_started":1}
I20260619 02:36:13.698869 19675 heartbeater.cc:347] Connected to a master server at 127.7.194.254:37817
I20260619 02:36:13.699146 19675 heartbeater.cc:464] Registering TS with master...
I20260619 02:36:13.699747 19675 heartbeater.cc:511] Master 127.7.194.254:37817 requested a full tablet report, sending...
I20260619 02:36:13.700989 19191 ts_manager.cc:194] Registered new tserver with Master: 67f5adc2ade44d0abf3798d124c43bd6 (127.7.194.195:45003)
I20260619 02:36:13.701741 19191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:39745
I20260619 02:36:13.707850 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:36:13.747443 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.722583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.7.194.254:37817)
0619 02:36:13.722736 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:13.722741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:13.728182 (+ 5441us) server_negotiation.cc:407] Connection header received
0619 02:36:13.728441 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:13.728446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:13.728511 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:13.728580 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:36:13.732997 (+ 4417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.733967 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.735417 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.735770 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.735873 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.740331 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:36:13.740337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:36:13.740341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:36:13.740384 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:36:13.742415 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:13.745686 (+ 3271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:13.745692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:13.745695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:13.745845 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:13.746289 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:13.746294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:13.746296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:13.746559 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:36:13.746675 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:13.747099 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 02:36:13.747258 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:36:13.752197 19191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:36:13.757144 19191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:36:13.792997 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.775125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:45003 (local address 127.0.0.1:45010)
0619 02:36:13.778614 (+ 3489us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:13.778630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:13.778731 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:13.782097 (+ 3366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:13.782101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:13.782118 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:36:13.782217 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.782224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.788739 (+ 6515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.788742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.789904 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.789917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.791331 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.791335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.792425 (+ 1090us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.792495 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 02:36:13.792636 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 02:36:13.792816 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3424,"thread_start_us":99,"threads_started":1}
I20260619 02:36:13.795185 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.771610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46703 (local address 127.0.0.1:45642)
0619 02:36:13.777844 (+ 6234us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:13.777914 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:13.778061 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:13.781514 (+ 3453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:13.781521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:13.781562 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:36:13.781721 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.781728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.785968 (+ 4240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.785973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.788302 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.788315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.794703 (+ 6388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.794708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.794819 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.794846 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 02:36:13.794932 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:36:13.795019 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5992,"thread_start_us":109,"threads_started":1}
I20260619 02:36:13.796394 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.775967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.7.194.195:45003)
0619 02:36:13.778462 (+ 2495us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:13.778476 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:13.778866 (+ 390us) server_negotiation.cc:407] Connection header received
0619 02:36:13.778965 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:13.778973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:13.779231 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:13.781942 (+ 2711us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:36:13.782359 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.788596 (+ 6237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.790095 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.790967 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.791088 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.791261 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:13.796076 (+ 4815us) server_negotiation.cc:299] Negotiation successful
0619 02:36:13.796223 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2374,"thread_start_us":124,"threads_started":1}
I20260619 02:36:13.798646 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.772804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.7.194.193:46703)
0619 02:36:13.780444 (+ 7640us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:13.780452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:13.780480 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:36:13.780569 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:13.780574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:13.780815 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:13.780994 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:36:13.783653 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.785752 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.793403 (+ 7651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.794337 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.794466 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.794632 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:13.798333 (+ 3701us) server_negotiation.cc:299] Negotiation successful
0619 02:36:13.798478 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7513,"thread_start_us":115,"threads_started":1}
I20260619 02:36:13.802956 19609 tablet_service.cc:1467] Processing CreateTablet for tablet a4b34ccec5d641aa9340e2fbac1fc50f (DEFAULT_TABLE table=test-table [id=bc99d38dc39546a39c40b3d7b0e66e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:36:13.804822 19609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b34ccec5d641aa9340e2fbac1fc50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:36:13.809520 19338 tablet_service.cc:1467] Processing CreateTablet for tablet a4b34ccec5d641aa9340e2fbac1fc50f (DEFAULT_TABLE table=test-table [id=bc99d38dc39546a39c40b3d7b0e66e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:36:13.810757 19338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b34ccec5d641aa9340e2fbac1fc50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:36:13.825268 19695 tablet_bootstrap.cc:492] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: Bootstrap starting.
I20260619 02:36:13.829154 19695 tablet_bootstrap.cc:654] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: Neither blocks nor log segments found. Creating new log.
I20260619 02:36:13.830343 19695 log.cc:824] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:36:13.832984 19695 tablet_bootstrap.cc:492] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: No bootstrap required, opened a new log
I20260619 02:36:13.833331 19695 ts_tablet_manager.cc:1397] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 02:36:13.838253 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.775473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:38481 (local address 127.0.0.1:48098)
0619 02:36:13.781263 (+ 5790us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:13.781282 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:13.781372 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:13.808072 (+ 26700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:13.808077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:13.808107 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:36:13.808238 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.808247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.816414 (+ 8167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.816419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.818212 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:13.818233 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.837702 (+ 19469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.837708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:13.837850 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.837888 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:36:13.837961 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:36:13.838064 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5724,"thread_start_us":91,"threads_started":1}
I20260619 02:36:13.839125 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:13.775667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.7.194.194:38481)
0619 02:36:13.796863 (+ 21196us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:13.796885 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:13.796918 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:36:13.797023 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:13.797031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:13.805871 (+ 8840us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:13.806065 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:36:13.814830 (+ 8765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.816204 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.818480 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:13.819512 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:13.838512 (+ 19000us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:13.838759 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:13.838832 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 02:36:13.838970 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21038,"thread_start_us":120,"threads_started":1}
I20260619 02:36:13.839895 19695 raft_consensus.cc:348] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:13.840391 19695 raft_consensus.cc:374] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:36:13.840940 19694 tablet_bootstrap.cc:492] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393: Bootstrap starting.
I20260619 02:36:13.841120 19695 raft_consensus.cc:729] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f5adc2ade44d0abf3798d124c43bd6, State: Initialized, Role: FOLLOWER
I20260619 02:36:13.841807 19695 consensus_queue.cc:260] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:13.842677 19456 tablet_service.cc:1467] Processing CreateTablet for tablet a4b34ccec5d641aa9340e2fbac1fc50f (DEFAULT_TABLE table=test-table [id=bc99d38dc39546a39c40b3d7b0e66e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:36:13.844079 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b34ccec5d641aa9340e2fbac1fc50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:36:13.844532 19695 ts_tablet_manager.cc:1428] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260619 02:36:13.845940 19675 heartbeater.cc:503] Master 127.7.194.254:37817 was elected leader, sending a full tablet report...
I20260619 02:36:13.851044 19694 tablet_bootstrap.cc:654] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393: Neither blocks nor log segments found. Creating new log.
I20260619 02:36:13.852106 19694 log.cc:824] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393: Log is configured to *not* fsync() on all Append() calls
I20260619 02:36:13.857766 19694 tablet_bootstrap.cc:492] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393: No bootstrap required, opened a new log
I20260619 02:36:13.858119 19694 ts_tablet_manager.cc:1397] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260619 02:36:13.863425 19694 raft_consensus.cc:348] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:13.863936 19694 raft_consensus.cc:374] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:36:13.864197 19694 raft_consensus.cc:729] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66092dfd11549afb9e109e2a1182393, State: Initialized, Role: FOLLOWER
I20260619 02:36:13.864912 19694 consensus_queue.cc:260] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:13.866243 19694 ts_tablet_manager.cc:1428] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 02:36:13.866927 19405 heartbeater.cc:503] Master 127.7.194.254:37817 was elected leader, sending a full tablet report...
I20260619 02:36:13.870193 19700 tablet_bootstrap.cc:492] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: Bootstrap starting.
I20260619 02:36:13.875322 19700 tablet_bootstrap.cc:654] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: Neither blocks nor log segments found. Creating new log.
I20260619 02:36:13.876919 19700 log.cc:824] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: Log is configured to *not* fsync() on all Append() calls
I20260619 02:36:13.882045 19700 tablet_bootstrap.cc:492] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: No bootstrap required, opened a new log
I20260619 02:36:13.882298 19700 ts_tablet_manager.cc:1397] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.002s
I20260619 02:36:13.888378 19700 raft_consensus.cc:348] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:13.888893 19700 raft_consensus.cc:374] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:36:13.889027 19700 raft_consensus.cc:729] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76bd5e516d9a4b88a10cdf5497993540, State: Initialized, Role: FOLLOWER
I20260619 02:36:13.889786 19700 consensus_queue.cc:260] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:13.891249 19700 ts_tablet_manager.cc:1428] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260619 02:36:13.891749 19541 heartbeater.cc:503] Master 127.7.194.254:37817 was elected leader, sending a full tablet report...
W20260619 02:36:13.922739 19406 tablet.cc:2369] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:36:13.926893 19676 tablet.cc:2369] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 02:36:14 dist-test-slave-9tkz krb5kdc[18715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836572, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
W20260619 02:36:14.056444 19542 tablet.cc:2369] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:36:14.193802 19697 raft_consensus.cc:482] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:36:14.194082 19697 raft_consensus.cc:504] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:14.195586 19697 leader_election.cc:288] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66092dfd11549afb9e109e2a1182393 (127.7.194.193:46703), 76bd5e516d9a4b88a10cdf5497993540 (127.7.194.194:38481)
I20260619 02:36:14.198693 19702 raft_consensus.cc:482] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:36:14.198904 19702 raft_consensus.cc:504] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:14.200268 19702 leader_election.cc:288] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66092dfd11549afb9e109e2a1182393 (127.7.194.193:46703), 67f5adc2ade44d0abf3798d124c43bd6 (127.7.194.195:45003)
I20260619 02:36:14.203477 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.196053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46703 (local address 127.7.194.195:42893)
0619 02:36:14.196839 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:14.196858 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:14.196971 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:14.197371 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:14.197376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:14.197401 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:36:14.197533 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:14.197574 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.200445 (+ 2871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.200449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:14.201871 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:14.201882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.202951 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.202962 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:14.203108 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:14.203134 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:36:14.203219 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:36:14.203311 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":22}
I20260619 02:36:14.204200 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.196802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:42893 (local address 127.7.194.193:46703)
0619 02:36:14.196935 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:14.196940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:14.196957 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:36:14.197081 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:14.197084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:14.197138 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:14.197215 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:36:14.197725 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.198492 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.202059 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.202692 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.202818 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:14.202875 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:14.203961 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0619 02:36:14.204062 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:36:14.204877 19343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b34ccec5d641aa9340e2fbac1fc50f" candidate_uuid: "67f5adc2ade44d0abf3798d124c43bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66092dfd11549afb9e109e2a1182393" is_pre_election: true
I20260619 02:36:14.205063 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.197247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:53203 (local address 127.7.194.194:38481)
0619 02:36:14.197379 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:14.197384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:14.197651 (+ 267us) server_negotiation.cc:407] Connection header received
0619 02:36:14.197861 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:14.197864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:14.197920 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:14.197974 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:36:14.198583 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.199282 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.202079 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.202692 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.202814 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:14.202875 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:14.204819 (+ 1944us) server_negotiation.cc:299] Negotiation successful
0619 02:36:14.204919 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:36:14.205278 19343 raft_consensus.cc:2454] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67f5adc2ade44d0abf3798d124c43bd6 in term 0.
I20260619 02:36:14.205622 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.196381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:38481 (local address 127.7.194.195:53203)
0619 02:36:14.197522 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:14.197536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:14.197721 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:14.198117 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:14.198120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:14.198137 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:36:14.198253 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:14.198261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.199444 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.199447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:14.201936 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:14.201947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.202993 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.202997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:14.204618 (+ 1621us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:14.204651 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:36:14.205287 (+ 636us) client_negotiation.cc:240] Negotiation successful
0619 02:36:14.205386 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260619 02:36:14.205981 19565 leader_election.cc:302] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67f5adc2ade44d0abf3798d124c43bd6, a66092dfd11549afb9e109e2a1182393; no voters:
I20260619 02:36:14.206504 19697 raft_consensus.cc:2781] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:36:14.206600 19697 raft_consensus.cc:482] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:36:14.206662 19697 raft_consensus.cc:3037] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:36:14.208242 19697 raft_consensus.cc:504] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:14.208725 19697 leader_election.cc:288] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [CANDIDATE]: Term 1 election: Requested vote from peers a66092dfd11549afb9e109e2a1182393 (127.7.194.193:46703), 76bd5e516d9a4b88a10cdf5497993540 (127.7.194.194:38481)
I20260619 02:36:14.209324 19477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b34ccec5d641aa9340e2fbac1fc50f" candidate_uuid: "67f5adc2ade44d0abf3798d124c43bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76bd5e516d9a4b88a10cdf5497993540" is_pre_election: true
I20260619 02:36:14.209602 19477 raft_consensus.cc:2454] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67f5adc2ade44d0abf3798d124c43bd6 in term 0.
I20260619 02:36:14.210991 19343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b34ccec5d641aa9340e2fbac1fc50f" candidate_uuid: "67f5adc2ade44d0abf3798d124c43bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66092dfd11549afb9e109e2a1182393"
I20260619 02:36:14.211148 19343 raft_consensus.cc:3037] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:36:14.211310 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.200412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46703 (local address 127.7.194.194:55321)
0619 02:36:14.205947 (+ 5535us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:14.205964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:14.206243 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:14.206690 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:14.206694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:14.206719 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:36:14.206911 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:14.206923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.207970 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.207973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:14.209466 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:14.209477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.210602 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.210606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:14.210719 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:14.210762 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 02:36:14.211024 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 02:36:14.211114 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5456,"thread_start_us":145,"threads_started":1}
I20260619 02:36:14.212713 19343 raft_consensus.cc:2454] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67f5adc2ade44d0abf3798d124c43bd6 in term 1.
I20260619 02:36:14.213706 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.201261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:55321 (local address 127.7.194.193:46703)
0619 02:36:14.204411 (+ 3150us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:14.204415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:14.206044 (+ 1629us) server_negotiation.cc:407] Connection header received
0619 02:36:14.206398 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:14.206404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:14.206470 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:14.206527 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:36:14.207068 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.207795 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.209658 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.210304 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.210407 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:14.210468 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:14.210901 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 02:36:14.210990 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3078,"thread_start_us":109,"threads_started":1}
I20260619 02:36:14.214507 19477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b34ccec5d641aa9340e2fbac1fc50f" candidate_uuid: "67f5adc2ade44d0abf3798d124c43bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76bd5e516d9a4b88a10cdf5497993540"
I20260619 02:36:14.214673 19477 raft_consensus.cc:3037] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:36:14.215888 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.201711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:58543 (local address 127.7.194.195:45003)
0619 02:36:14.201852 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:14.201857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:14.211570 (+ 9713us) server_negotiation.cc:407] Connection header received
0619 02:36:14.211780 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:14.211783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:14.211840 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:14.211892 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:36:14.212277 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.213038 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.214496 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.215116 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.215218 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:14.215281 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:14.215627 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 02:36:14.215726 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:36:14.216178 19477 raft_consensus.cc:2454] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67f5adc2ade44d0abf3798d124c43bd6 in term 1.
I20260619 02:36:14.216320 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.200890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:45003 (local address 127.7.194.194:58543)
0619 02:36:14.211482 (+ 10592us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:14.211501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:36:14.211644 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:36:14.212034 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:36:14.212037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:36:14.212054 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:36:14.212140 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:14.212146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.213190 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.213194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:14.214341 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:36:14.214352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.215385 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.215389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:36:14.215480 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:14.215504 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:36:14.216095 (+ 591us) client_negotiation.cc:240] Negotiation successful
0619 02:36:14.216182 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10510,"thread_start_us":104,"threads_started":1}
I20260619 02:36:14.216939 19565 leader_election.cc:302] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67f5adc2ade44d0abf3798d124c43bd6, 76bd5e516d9a4b88a10cdf5497993540; no voters:
I20260619 02:36:14.217255 19614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b34ccec5d641aa9340e2fbac1fc50f" candidate_uuid: "76bd5e516d9a4b88a10cdf5497993540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f5adc2ade44d0abf3798d124c43bd6" is_pre_election: true
I20260619 02:36:14.217339 19343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b34ccec5d641aa9340e2fbac1fc50f" candidate_uuid: "76bd5e516d9a4b88a10cdf5497993540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66092dfd11549afb9e109e2a1182393" is_pre_election: true
I20260619 02:36:14.217535 19343 raft_consensus.cc:2379] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76bd5e516d9a4b88a10cdf5497993540 in current term 1: Already voted for candidate 67f5adc2ade44d0abf3798d124c43bd6 in this term.
I20260619 02:36:14.217537 19614 raft_consensus.cc:2379] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76bd5e516d9a4b88a10cdf5497993540 in current term 1: Already voted for candidate 67f5adc2ade44d0abf3798d124c43bd6 in this term.
I20260619 02:36:14.218647 19431 leader_election.cc:302] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76bd5e516d9a4b88a10cdf5497993540; no voters: 67f5adc2ade44d0abf3798d124c43bd6, a66092dfd11549afb9e109e2a1182393
I20260619 02:36:14.219209 19702 raft_consensus.cc:2726] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 02:36:14.219513 19697 raft_consensus.cc:2781] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:36:14.219764 19697 raft_consensus.cc:686] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 1 LEADER]: Becoming Leader. State: Replica: 67f5adc2ade44d0abf3798d124c43bd6, State: Running, Role: LEADER
I20260619 02:36:14.220150 19697 consensus_queue.cc:237] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } }
I20260619 02:36:14.224457 19191 catalog_manager.cc:5495] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67f5adc2ade44d0abf3798d124c43bd6 (127.7.194.195). New cstate: current_term: 1 leader_uuid: "67f5adc2ade44d0abf3798d124c43bd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67f5adc2ade44d0abf3798d124c43bd6" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20260619 02:36:14.239634 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.235448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.7.194.195:45003)
0619 02:36:14.235574 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:14.235579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:14.235663 (+ 84us) server_negotiation.cc:407] Connection header received
0619 02:36:14.235867 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:14.235870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:14.235930 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:14.236050 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:36:14.236291 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.237060 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.238078 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:14.238321 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:14.238432 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:14.238763 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:36:14.239227 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:36:14.239363 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:14.239398 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 02:36:14.239458 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:36:14.246059 19477 raft_consensus.cc:1261] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 1 FOLLOWER]: Refusing update from remote peer 67f5adc2ade44d0abf3798d124c43bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:36:14.246160 19343 raft_consensus.cc:1261] T a4b34ccec5d641aa9340e2fbac1fc50f P a66092dfd11549afb9e109e2a1182393 [term 1 FOLLOWER]: Refusing update from remote peer 67f5adc2ade44d0abf3798d124c43bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:36:14.246866 19709 consensus_queue.cc:1035] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66092dfd11549afb9e109e2a1182393" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:36:14.247332 19709 consensus_queue.cc:1035] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76bd5e516d9a4b88a10cdf5497993540" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:36:14.265383 19712 mvcc.cc:205] Tried to move back new op lower bound from 7298402608103526400 to 7298402608010870784. Current Snapshot: MvccSnapshot[applied={T|T < 7298402608103526400}]
I20260619 02:36:14.270051 19713 mvcc.cc:205] Tried to move back new op lower bound from 7298402608103526400 to 7298402608010870784. Current Snapshot: MvccSnapshot[applied={T|T < 7298402608103526400}]
I20260619 02:36:14.271649 19715 mvcc.cc:205] Tried to move back new op lower bound from 7298402608103526400 to 7298402608010870784. Current Snapshot: MvccSnapshot[applied={T|T < 7298402608103526400}]
I20260619 02:36:14.279361 19191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:36:14.279569 19191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:36:14.282109 19191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6378f373ddf7473a9eea370277c6a4cf: Sending DeleteTablet for 3 replicas of tablet a4b34ccec5d641aa9340e2fbac1fc50f
I20260619 02:36:14.283119 19456 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b34ccec5d641aa9340e2fbac1fc50f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:36:14 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:48098
I20260619 02:36:14.283135 19338 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b34ccec5d641aa9340e2fbac1fc50f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:36:14 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:45642
I20260619 02:36:14.283707 19729 tablet_replica.cc:330] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: stopping tablet replica
I20260619 02:36:14.283736 19609 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b34ccec5d641aa9340e2fbac1fc50f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:36:14 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:45010
I20260619 02:36:14.284157 19729 raft_consensus.cc:2229] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:36:14.284623 19731 tablet_replica.cc:330] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: stopping tablet replica
I20260619 02:36:14.285043 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 19272
I20260619 02:36:14.285111 19729 raft_consensus.cc:2258] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:36:14.287581 19731 raft_consensus.cc:2229] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:36:14.287974 19729 ts_tablet_manager.cc:1905] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:36:14.288182 19731 raft_consensus.cc:2258] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:36:14.289716 19731 ts_tablet_manager.cc:1905] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:36:14.291061 19729 ts_tablet_manager.cc:1918] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:36:14.291169 19729 log.cc:1196] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/a4b34ccec5d641aa9340e2fbac1fc50f
I20260619 02:36:14.291510 19729 ts_tablet_manager.cc:1939] T a4b34ccec5d641aa9340e2fbac1fc50f P 76bd5e516d9a4b88a10cdf5497993540: Deleting consensus metadata
I20260619 02:36:14.292663 19731 ts_tablet_manager.cc:1918] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:36:14.292830 19731 log.cc:1196] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/a4b34ccec5d641aa9340e2fbac1fc50f
I20260619 02:36:14.292917 19178 catalog_manager.cc:4841] TS 76bd5e516d9a4b88a10cdf5497993540 (127.7.194.194:38481): tablet a4b34ccec5d641aa9340e2fbac1fc50f (table test-table [id=bc99d38dc39546a39c40b3d7b0e66e0d]) successfully deleted
I20260619 02:36:14.293354 19731 ts_tablet_manager.cc:1939] T a4b34ccec5d641aa9340e2fbac1fc50f P 67f5adc2ade44d0abf3798d124c43bd6: Deleting consensus metadata
I20260619 02:36:14.294840 19178 catalog_manager.cc:4841] TS 67f5adc2ade44d0abf3798d124c43bd6 (127.7.194.195:45003): tablet a4b34ccec5d641aa9340e2fbac1fc50f (table test-table [id=bc99d38dc39546a39c40b3d7b0e66e0d]) successfully deleted
W20260619 02:36:14.298902 19178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:46703 (error 108)
I20260619 02:36:14.299525 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 19411
I20260619 02:36:14.306700 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:14.306357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:46703 (local address 127.0.0.1:45654)
0619 02:36:14.306498 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 02:36:14.306581 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 02:36:14.307021 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 19546
W20260619 02:36:14.307089 19178 catalog_manager.cc:4584] TS a66092dfd11549afb9e109e2a1182393 (127.7.194.193:46703): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4b34ccec5d641aa9340e2fbac1fc50f: Network error: Client connection negotiation failed: client connection to 127.7.194.193:46703: connect: Connection refused (error 111)
I20260619 02:36:14.312808 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 19159
I20260619 02:36:14.319742 7947 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:36:14.665707 7947 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:36:14.665882 7947 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:36:15.011595 7947 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:36:15.011 UTC [18745] LOG: received smart shutdown request
2026-06-19 02:36:15.015 UTC [18745] LOG: background worker "logical replication launcher" (PID 18754) exited with exit code 1
2026-06-19 02:36:15.016 UTC [18749] LOG: shutting down
2026-06-19 02:36:15.178 UTC [18745] LOG: database system is shut down
2026-06-19T02:36:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48528 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:36:15 dist-test-slave-9tkz krb5kdc[19741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:36:15 dist-test-slave-9tkz krb5kdc[19741](info): set up 2 sockets
Jun 19 02:36:15 dist-test-slave-9tkz krb5kdc[19741](info): commencing operation
krb5kdc: starting...
W20260619 02:36:16.374637 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:36:16 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:36:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/rangeradmin_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/rangeradmin_127.7.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/rangerlookup_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/rangerlookup_127.7.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/HTTP_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/HTTP_127.7.194.212@KRBTEST.COM.keytab.
I20260619 02:36:16.549227 7947 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task73xKWu/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/postgres -l logfile start
2026-06-19 02:36:19.065 UTC [19771] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:36:19.065 UTC [19771] LOG: listening on IPv4 address "127.7.194.212", port 35255
2026-06-19 02:36:19.069 UTC [19771] LOG: listening on Unix socket "/tmp/.s.PGSQL.35255"
2026-06-19 02:36:19.089 UTC [19774] LOG: database system was shut down at 2026-06-19 02:36:17 UTC
2026-06-19 02:36:19.094 UTC [19771] LOG: database system is ready to accept connections
I20260619 02:36:20.089709 7947 mini_postgres.cc:96] Postgres bound to 35255
2026-06-19 02:36:20.096 UTC [19782] FATAL: database "slave" does not exist
127.7.194.212:35255 - accepting connections
I20260619 02:36:20.096928 7947 mini_ranger.cc:161] Starting Ranger...
I20260619 02:36:20.112445 7947 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:36:20.617708 7947 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:36:20.617796 7947 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-admin
2026-06-19 02:36:20,909 [I] DB FLAVOR :POSTGRES
2026-06-19 02:36:20,909 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:36:20,910 [I] Checking connection..
2026-06-19 02:36:20,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:36:21,181 [I] Checking connection passed.
2026-06-19 02:36:21,181 [I] --------- Verifying version history table ---------
2026-06-19 02:36:21,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:36:21,429 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:36:21,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:36:21,688 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:36:21,689 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:36:21,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:36:21.956 UTC [19846] WARNING: there is no transaction in progress
2026-06-19 02:36:21,963 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:36:21,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:36:22,212 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:36:22,212 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:36:22,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:36:22,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:36:22,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9tkz.c.gcp-upstream.internal','N') ;"
2026-06-19 02:36:23,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:36:23,283 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:36:23,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:36:23,552 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:36:23,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:36:23,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:36:24,045 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:36:24,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:36:25.036 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.054 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.073 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.092 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.110 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.128 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.188 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.196 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.213 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.224 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.235 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.250 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.261 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.268 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.277 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25.285 UTC [19954] WARNING: there is no transaction in progress
2026-06-19 02:36:25,985 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:36:25,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:36:26,255 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:36:26,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:36:26,499 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:36:26,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:36:26,744 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:36:26,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:36:26,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:36:27,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9tkz.c.gcp-upstream.internal';"
2026-06-19 02:36:27,477 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:36:27,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:36:27,729 [I] DB_PATCHES have already been applied
I20260619 02:36:27.735535 7947 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-admin:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task73xKWu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:36:27.735616 7947 mini_ranger.cc:193] Using host: 127.7.194.212
I20260619 02:36:27.739210 7947 mini_ranger.cc:227] Ranger admin URL: http://127.7.194.212:37857
Jun 19, 2026 2:36:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:36:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:36:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:36:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:36:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:36:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:36:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:36:28.840929 7947 mini_ranger.cc:160] Time spent starting Ranger: real 8.744s user 0.000s sys 0.010s
Jun 19, 2026 2:36:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:36:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 2:36:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:36:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:36:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:36:50.920431 7947 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/rangerkms_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/rangerkms_127.7.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.194.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.194.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.194.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/HTTP_127.7.194.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.194.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/HTTP_127.7.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:36:51 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836611, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:36:51.006778 7947 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:36:51.016707 7947 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:36:51.852902 7947 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:36:51.852989 7947 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-kms
I20260619 02:36:51.853012 7947 mini_ranger_kms.cc:227] Using postgres at 127.7.194.212:35255
2026-06-19 02:36:52,029 [I] DB FLAVOR :POSTGRES
2026-06-19 02:36:52,029 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:36:52,029 [I] Checking connection
2026-06-19 02:36:52,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:36:52,260 [I] connection success
2026-06-19 02:36:52,261 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:36:52,261 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:36:52,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:36:52,497 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:36:52,497 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:36:52,498 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:36:52,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.194.212:35255/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:36:52,774 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:36:53.229003 7947 mini_ranger_kms.cc:310] Created kms service
I20260619 02:36:53.422427 7947 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:36:53.469435 7947 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:36:53 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836611, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:36:53.888826 7947 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:36:53.888993 7947 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-kms:/tmp/dist-test-task73xKWu/build/debug/bin/postgresql.jar:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task73xKWu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:36:53.889024 7947 mini_ranger_kms.cc:240] Using host: 127.7.194.212
I20260619 02:36:53.892802 7947 mini_ranger_kms.cc:276] Ranger KMS PID: 20134
I20260619 02:36:53.892896 7947 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.194.212:58455
Jun 19, 2026 2:36:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:36:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:36:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:36:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:36:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:36:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 02:36:56.129534 7947 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.123s user 0.014s sys 0.147s
I20260619 02:36:56.129695 7947 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:36:56.129773 7947 mini_ranger_kms.cc:401] 127.7.194.212:58455/kms/v1/keys
Jun 19 02:36:56 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781836616, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.194.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 02:36:56 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836616, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.194.212@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
Jun 19 02:36:57 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836616, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.194.212@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
Jun 19 02:36:59 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:36:59.330894 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:42857
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:42857
--ranger_config_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:58455/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:36:59.446380 20185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:36:59.446794 20185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:36:59.446933 20185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:36:59.451184 20185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:36:59.451294 20185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:36:59.451331 20185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:36:59.451375 20185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:36:59.451408 20185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:36:59.457311 20185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.194.212:58455/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:42857
--ranger_config_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:42857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:36:59.458518 20185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:36:59.459632 20185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:36:59.468961 20193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:36:59.468961 20191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:36:59.469013 20185 server_base.cc:1034] running on GCE node
W20260619 02:36:59.468962 20190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:36:59.469946 20185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:36:59.471089 20185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:36:59.472288 20185 hybrid_clock.cc:648] HybridClock initialized: now 1781836619472261 us; error 39 us; skew 500 ppm
Jun 19 02:36:59 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:36:59.475287 20185 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:36:59.476191 20185 webserver.cc:460] Webserver started at http://127.7.194.254:37739/ using document root <none> and password file <none>
I20260619 02:36:59.476791 20185 fs_manager.cc:359] Metadata directory not provided
I20260619 02:36:59.476873 20185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:36:59.477108 20185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:36:59 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:36:59.533470 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "5948d44be97140d1a2529fa4f84e7dc9"
format_stamp: "Formatted at 2026-06-19 02:36:59 on dist-test-slave-9tkz"
server_key: "161d6ea19d31cde7398f46324ba225da"
server_key_iv: "93d499eec7d65f51fc888d7dca5d320a"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:59.534340 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "5948d44be97140d1a2529fa4f84e7dc9"
format_stamp: "Formatted at 2026-06-19 02:36:59 on dist-test-slave-9tkz"
server_key: "161d6ea19d31cde7398f46324ba225da"
server_key_iv: "93d499eec7d65f51fc888d7dca5d320a"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:59.539752 20185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:36:59.558058 20200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:36:59.559617 20185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:36:59.559768 20185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "5948d44be97140d1a2529fa4f84e7dc9"
format_stamp: "Formatted at 2026-06-19 02:36:59 on dist-test-slave-9tkz"
server_key: "161d6ea19d31cde7398f46324ba225da"
server_key_iv: "93d499eec7d65f51fc888d7dca5d320a"
server_key_version: "kuduclusterkey@0"
I20260619 02:36:59.559913 20185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:36:59.572201 20185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:36:59.575501 20185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:36:59.575714 20185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:36:59.583539 20185 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:42857
I20260619 02:36:59.583575 20252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:42857 every 8 connection(s)
I20260619 02:36:59.584618 20185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:36:59.587675 20253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:36:59.589423 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 20185
I20260619 02:36:59.589627 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:36:59.589980 7947 external_mini_cluster.cc:1428] Setting key 3c37448bb71be7cd13a56c1861880ff0
I20260619 02:36:59.594762 20253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Bootstrap starting.
I20260619 02:36:59.597186 20253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Neither blocks nor log segments found. Creating new log.
I20260619 02:36:59.598189 20253 log.cc:824] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:36:59 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:36:59.600558 20253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: No bootstrap required, opened a new log
I20260619 02:36:59.604127 20253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } }
I20260619 02:36:59.604372 20253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:36:59.604496 20253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5948d44be97140d1a2529fa4f84e7dc9, State: Initialized, Role: FOLLOWER
I20260619 02:36:59.605053 20253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } }
I20260619 02:36:59.605192 20253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:36:59.605284 20253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:36:59.605396 20253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:36:59.605846 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:36:59.591690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.7.194.254:42857)
0619 02:36:59.592158 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 02:36:59.592182 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 02:36:59.592220 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:36:59.592906 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:36:59.592923 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:36:59.593307 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:36:59.593717 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:36:59.594564 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:59.595611 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:59.596523 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:36:59.597004 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:36:59.597184 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:36:59.600151 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:36:59.600157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:36:59.600171 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:36:59.600197 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:36:59.602745 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:59.603207 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:59.603217 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:59.603221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:59.603326 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:36:59.603609 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:36:59.603614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:36:59.603615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:36:59.604027 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:36:59.604293 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 02:36:59.604555 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:36:59.604861 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":143,"threads_started":1}
I20260619 02:36:59.606527 20253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } }
I20260619 02:36:59.606907 20253 leader_election.cc:302] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5948d44be97140d1a2529fa4f84e7dc9; no voters:
I20260619 02:36:59.607234 20253 leader_election.cc:288] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:36:59.607365 20258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:36:59.607561 20258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 1 LEADER]: Becoming Leader. State: Replica: 5948d44be97140d1a2529fa4f84e7dc9, State: Running, Role: LEADER
I20260619 02:36:59.607929 20258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } }
I20260619 02:36:59.608690 20253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:36:59.609157 20258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5948d44be97140d1a2529fa4f84e7dc9. Latest consensus state: current_term: 1 leader_uuid: "5948d44be97140d1a2529fa4f84e7dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } } }
I20260619 02:36:59.609347 20258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:36:59.609709 20259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5948d44be97140d1a2529fa4f84e7dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } } }
I20260619 02:36:59.609913 20259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:36:59.609810 20263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:36:59.613272 20263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:36:59.613718 20253 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:36:59.623175 20263 catalog_manager.cc:1347] Generated new cluster ID: a62624cfc8cd4d8ba3f9f702b441fc4e
I20260619 02:36:59.623256 20263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:36:59.636765 20263 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:36:59.638124 20263 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:36:59.648752 20263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Generated new TSK 0
I20260619 02:36:59.649669 20263 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:37:00.340044 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:58455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42857
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:00.496261 20298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:00.496598 20298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:00.496686 20298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:00.501036 20298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:00.501147 20298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:00.501250 20298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:37:00.507298 20298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.194.212:58455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:00.508586 20298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:00.510001 20298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:00.522153 20308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:00.522204 20306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:00.522166 20305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:00.522783 20298 server_base.cc:1034] running on GCE node
I20260619 02:37:00.523316 20298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:00.524135 20298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:00.525377 20298 hybrid_clock.cc:648] HybridClock initialized: now 1781836620525336 us; error 54 us; skew 500 ppm
Jun 19 02:37:00 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:00.528728 20298 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:37:00.530145 20298 webserver.cc:460] Webserver started at http://127.7.194.193:42305/ using document root <none> and password file <none>
I20260619 02:37:00.531049 20298 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:00.531154 20298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:00.531414 20298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:37:00 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:00.550549 20298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "1bdfb0eb440c46bdb913508abfea9e4b"
format_stamp: "Formatted at 2026-06-19 02:37:00 on dist-test-slave-9tkz"
server_key: "028666708506e7891d8921168abc3aae"
server_key_iv: "fdb9f0dadd603123a3887a83fba2f670"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:00.551375 20298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "1bdfb0eb440c46bdb913508abfea9e4b"
format_stamp: "Formatted at 2026-06-19 02:37:00 on dist-test-slave-9tkz"
server_key: "028666708506e7891d8921168abc3aae"
server_key_iv: "fdb9f0dadd603123a3887a83fba2f670"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:00.557003 20298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 02:37:00.576357 20315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:00.578385 20298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:37:00.578540 20298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "1bdfb0eb440c46bdb913508abfea9e4b"
format_stamp: "Formatted at 2026-06-19 02:37:00 on dist-test-slave-9tkz"
server_key: "028666708506e7891d8921168abc3aae"
server_key_iv: "fdb9f0dadd603123a3887a83fba2f670"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:00.578692 20298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:00.604521 20298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:00.608476 20298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:00.608759 20298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:00.609647 20298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:00.610929 20298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:00.611032 20298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:00.611161 20298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:00.611253 20298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:00.624713 20298 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:33045
I20260619 02:37:00.626109 20298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:37:00.626991 20428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:33045 every 8 connection(s)
I20260619 02:37:00.636302 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 20298
I20260619 02:37:00.636467 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:37:00.636807 7947 external_mini_cluster.cc:1428] Setting key 28ac4c5aaf2ccda337a30b3ca0961084
Jun 19 02:37:00 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:00.645741 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:00.631202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:37331 (local address 127.7.194.254:42857)
0619 02:37:00.633429 (+ 2227us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:00.633438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:00.633453 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:37:00.633498 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:00.633502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:00.633586 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:00.633673 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:00.635319 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:00.636185 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:00.638326 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:00.638644 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:00.638781 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:00.641402 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:00.641405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:00.641408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:00.641432 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:00.643312 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:00.643912 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:00.643917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:00.643919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:00.644015 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:00.644417 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:00.644421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:00.644423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:00.644644 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:00.644759 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:00.645322 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 02:37:00.645462 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2145,"thread_start_us":112,"threads_started":1}
I20260619 02:37:00.646680 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:00.629966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:42857 (local address 127.7.194.193:37331)
0619 02:37:00.631803 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:00.631868 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:00.632974 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:00.634032 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:00.634042 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:00.634416 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:00.635141 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:00.635161 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:00.636348 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:00.636351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:00.638057 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:00.638070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:00.638836 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:00.638840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:00.639039 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:00.639698 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:00.639717 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:00.641194 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:00.643461 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:00.643468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:00.643485 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:00.643780 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:00.644123 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:00.644128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:00.644130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:00.644302 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:00.644771 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:00.644779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:00.645055 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 02:37:00.645279 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 02:37:00.645664 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1567,"thread_start_us":168,"threads_started":1}
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:37:00.648228 20429 heartbeater.cc:347] Connected to a master server at 127.7.194.254:42857
I20260619 02:37:00.648512 20429 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:00.649065 20429 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:37:00.651029 20217 ts_manager.cc:194] Registered new tserver with Master: 1bdfb0eb440c46bdb913508abfea9e4b (127.7.194.193:33045)
I20260619 02:37:00.652669 20217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:37331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:37:00.668720 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:58455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42857
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:37:00 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781836620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 02:37:00.827867 20437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:00.828267 20437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:00.828339 20437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:00.834275 20437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:00.834369 20437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:00.834466 20437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:37:00.842777 20437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.194.212:58455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:00.844264 20437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:00.845686 20437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:00.859445 20446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:00.859665 20444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:00.859930 20443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:00.860172 20437 server_base.cc:1034] running on GCE node
I20260619 02:37:00.860713 20437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:00.861646 20437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:00.862926 20437 hybrid_clock.cc:648] HybridClock initialized: now 1781836620862891 us; error 52 us; skew 500 ppm
Jun 19 02:37:00 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:00.867056 20437 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:37:00.868281 20437 webserver.cc:460] Webserver started at http://127.7.194.194:42927/ using document root <none> and password file <none>
I20260619 02:37:00.869102 20437 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:00.869248 20437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:00.869581 20437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:37:00 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:00.886551 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "7485526f008349bbaa9e6933fc3035ce"
format_stamp: "Formatted at 2026-06-19 02:37:00 on dist-test-slave-9tkz"
server_key: "5bb9afd060495ff6845ceb852c539c34"
server_key_iv: "2dc29ddd79159c35b97085a26433f1ad"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:00.887578 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "7485526f008349bbaa9e6933fc3035ce"
format_stamp: "Formatted at 2026-06-19 02:37:00 on dist-test-slave-9tkz"
server_key: "5bb9afd060495ff6845ceb852c539c34"
server_key_iv: "2dc29ddd79159c35b97085a26433f1ad"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:00.895512 20437 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260619 02:37:00.923128 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:00.925112 20437 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:37:00.925266 20437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "7485526f008349bbaa9e6933fc3035ce"
format_stamp: "Formatted at 2026-06-19 02:37:00 on dist-test-slave-9tkz"
server_key: "5bb9afd060495ff6845ceb852c539c34"
server_key_iv: "2dc29ddd79159c35b97085a26433f1ad"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:00.925422 20437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:00.942245 20437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:00.946388 20437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:00.946677 20437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:00.947494 20437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:00.948676 20437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:00.948747 20437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:00.948807 20437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:00.948900 20437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 02:37:00.960930 20437 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:38389
I20260619 02:37:00.961001 20566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:38389 every 8 connection(s)
I20260619 02:37:00.962368 20437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:37:00.968451 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 20437
I20260619 02:37:00.968603 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:37:00.968930 7947 external_mini_cluster.cc:1428] Setting key 719385fa4a6375dcae76c1af0679b61e
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
Jun 19 02:37:00 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:00.992208 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:00.965741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:32787 (local address 127.7.194.254:42857)
0619 02:37:00.969688 (+ 3947us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:00.969693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:00.969712 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:37:00.969758 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:00.969761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:00.969816 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:00.969911 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:00.978506 (+ 8595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:00.979377 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:00.983819 (+ 4442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:00.984293 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:00.984387 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:00.987677 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:00.987681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:00.987684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:00.987714 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:00.989615 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:00.990290 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:00.990294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:00.990296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:00.990386 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:00.990822 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:00.990826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:00.990828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:00.991060 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:00.991144 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:00.991869 (+ 725us) server_negotiation.cc:299] Negotiation successful
0619 02:37:00.992025 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3834}
I20260619 02:37:00.994074 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:00.965075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:42857 (local address 127.7.194.194:32787)
0619 02:37:00.966649 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:00.966710 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:00.967803 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:00.976790 (+ 8987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:00.976804 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:00.977294 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:00.978188 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:00.978216 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:00.979577 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:00.979580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:00.983609 (+ 4029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:00.983624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:00.984629 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:00.984633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:00.984869 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:00.985625 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:00.985649 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:00.987438 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:00.989766 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:00.989773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:00.989794 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:00.990161 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:00.990507 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:00.990510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:00.990512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:00.990699 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:00.991212 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:00.991219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:00.991557 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 02:37:00.992459 (+ 902us) client_negotiation.cc:240] Negotiation successful
0619 02:37:00.992856 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":140,"threads_started":1}
I20260619 02:37:00.995837 20567 heartbeater.cc:347] Connected to a master server at 127.7.194.254:42857
I20260619 02:37:00.996130 20567 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:00.996784 20567 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
I20260619 02:37:00.998067 20217 ts_manager.cc:194] Registered new tserver with Master: 7485526f008349bbaa9e6933fc3035ce (127.7.194.194:38389)
I20260619 02:37:00.998770 20217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:32787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:37:01.006798 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:58455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42857
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:01.167292 20572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:01.167676 20572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:01.167812 20572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:01.174001 20572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:01.174167 20572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:01.174320 20572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:37:01.182503 20572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.194.212:58455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:42857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:01.184393 20572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:01.185945 20572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:01.199120 20580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:01.200389 20578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:01.200945 20577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:01.201419 20572 server_base.cc:1034] running on GCE node
I20260619 02:37:01.201942 20572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
Jun 19 02:37:01 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:01.202697 20572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:01.205967 20572 hybrid_clock.cc:648] HybridClock initialized: now 1781836621205887 us; error 87 us; skew 500 ppm
Jun 19 02:37:01 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:01.210549 20572 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:37:01.211807 20572 webserver.cc:460] Webserver started at http://127.7.194.195:43061/ using document root <none> and password file <none>
I20260619 02:37:01.212569 20572 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:01.212708 20572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:01.213008 20572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:37:01 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:01.242980 20572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "48255ac8f863483c85c8efdd4c09e1db"
format_stamp: "Formatted at 2026-06-19 02:37:01 on dist-test-slave-9tkz"
server_key: "16c15b066b5adfbb86e0cf47472f29b9"
server_key_iv: "bcd40c61d146ecd2aaeaee519ee1a2b6"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:01.243814 20572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "48255ac8f863483c85c8efdd4c09e1db"
format_stamp: "Formatted at 2026-06-19 02:37:01 on dist-test-slave-9tkz"
server_key: "16c15b066b5adfbb86e0cf47472f29b9"
server_key_iv: "bcd40c61d146ecd2aaeaee519ee1a2b6"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:01.251276 20572 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 02:37:01.276407 20587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:01.283982 20572 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260619 02:37:01.284148 20572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "48255ac8f863483c85c8efdd4c09e1db"
format_stamp: "Formatted at 2026-06-19 02:37:01 on dist-test-slave-9tkz"
server_key: "16c15b066b5adfbb86e0cf47472f29b9"
server_key_iv: "bcd40c61d146ecd2aaeaee519ee1a2b6"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:01.284294 20572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:01.305687 20572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:01.309971 20572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:01.310226 20572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:01.311074 20572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:01.312171 20572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:01.312285 20572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:01.312383 20572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:01.312443 20572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:01.322419 20572 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:39215
I20260619 02:37:01.323510 20700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:39215 every 8 connection(s)
I20260619 02:37:01.323712 20572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:37:01.324796 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 20572
I20260619 02:37:01.324939 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:37:01.325302 7947 external_mini_cluster.cc:1428] Setting key 3ceb712c4170f591accae56d6d050393
Jun 19 02:37:01 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:01.343604 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:01.327036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:45599 (local address 127.7.194.254:42857)
0619 02:37:01.327156 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:01.327161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:01.328326 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 02:37:01.329251 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:01.329254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:01.329308 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:01.329389 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:01.330635 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:01.331240 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:01.332529 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:01.332769 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:01.332882 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:01.339857 (+ 6975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:01.339863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:01.339871 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:01.339896 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:01.341428 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:01.342014 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:01.342021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:01.342023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:01.342106 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:01.342466 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:01.342469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:01.342471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:01.342669 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:01.342776 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:01.343313 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 02:37:01.343443 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:37:01.344677 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:01.327648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:42857 (local address 127.7.194.195:45599)
0619 02:37:01.328180 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:01.328224 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:01.329049 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:01.329617 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:01.329630 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:01.329908 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:01.330476 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:01.330498 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:01.331372 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:01.331375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:01.332396 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:01.332405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:01.332919 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:01.332923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:01.333093 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:01.337874 (+ 4781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:01.337897 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:01.339688 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:01.341593 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:01.341600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:01.341618 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:01.341906 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:01.342201 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:01.342205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:01.342207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:01.342363 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:01.342774 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:01.342782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:01.343062 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 02:37:01.343262 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 02:37:01.343603 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":144,"threads_started":1}
I20260619 02:37:01.345890 20701 heartbeater.cc:347] Connected to a master server at 127.7.194.254:42857
I20260619 02:37:01.346167 20701 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:01.346694 20701 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
I20260619 02:37:01.347786 20217 ts_manager.cc:194] Registered new tserver with Master: 48255ac8f863483c85c8efdd4c09e1db (127.7.194.195:39215)
I20260619 02:37:01.348372 20217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:45599
I20260619 02:37:01.351683 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:37:01.352370 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 20298
I20260619 02:37:01.358081 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 20437
I20260619 02:37:01.364880 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 20572
I20260619 02:37:01.373391 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 20185
I20260619 02:37:01.379294 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:42857
--webserver_interface=127.7.194.254
--webserver_port=37739
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:42857
--ranger_config_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:58455/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.194.212:58455/kms with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:01.495781 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:01.496104 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:01.496192 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:01.500401 20704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:01.500509 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:01.500548 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:01.500586 20704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:01.500626 20704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:01.506479 20704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.194.212:58455/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:42857
--ranger_config_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:42857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:01.507620 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:01.508704 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:01.517522 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:01.517679 20704 server_base.cc:1034] running on GCE node
W20260619 02:37:01.517853 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:01.517516 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:01.518503 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:01.519644 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:01.520859 20704 hybrid_clock.cc:648] HybridClock initialized: now 1781836621520836 us; error 43 us; skew 500 ppm
Jun 19 02:37:01 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:01.524111 20704 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:37:01.525214 20704 webserver.cc:460] Webserver started at http://127.7.194.254:37739/ using document root <none> and password file <none>
I20260619 02:37:01.525907 20704 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:01.526008 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:37:01 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:01.607918 20704 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 02:37:01.610807 20720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:01.612447 20704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:37:01.612591 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "5948d44be97140d1a2529fa4f84e7dc9"
format_stamp: "Formatted at 2026-06-19 02:36:59 on dist-test-slave-9tkz"
server_key: "161d6ea19d31cde7398f46324ba225da"
server_key_iv: "93d499eec7d65f51fc888d7dca5d320a"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:01.613358 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:01.645375 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:01.648319 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:01.648488 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:01.655912 20704 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:42857
I20260619 02:37:01.655941 20772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:42857 every 8 connection(s)
I20260619 02:37:01.657054 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:37:01.657850 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 20704
I20260619 02:37:01.658387 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:58455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:33045
--local_ip_for_outbound_sockets=127.7.194.193
--tserver_master_addrs=127.7.194.254:42857
--webserver_port=42305
--webserver_interface=127.7.194.193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.194.212:58455/kms with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:01.662606 20773 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:37:01.664337 20773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Bootstrap starting.
I20260619 02:37:01.681051 20773 log.cc:824] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:01.687853 20773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:37:01.688449 20773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Bootstrap complete.
I20260619 02:37:01.692718 20773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } }
I20260619 02:37:01.693176 20773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5948d44be97140d1a2529fa4f84e7dc9, State: Initialized, Role: FOLLOWER
I20260619 02:37:01.693737 20773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } }
I20260619 02:37:01.693853 20773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:37:01.693941 20773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:37:01.694033 20773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:37:01.695998 20773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } }
I20260619 02:37:01.696353 20773 leader_election.cc:302] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5948d44be97140d1a2529fa4f84e7dc9; no voters:
I20260619 02:37:01.696732 20773 leader_election.cc:288] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:37:01.696822 20777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:37:01.697088 20777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [term 2 LEADER]: Becoming Leader. State: Replica: 5948d44be97140d1a2529fa4f84e7dc9, State: Running, Role: LEADER
I20260619 02:37:01.697453 20777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } }
I20260619 02:37:01.698041 20773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:37:01.698958 20779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5948d44be97140d1a2529fa4f84e7dc9. Latest consensus state: current_term: 2 leader_uuid: "5948d44be97140d1a2529fa4f84e7dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } } }
I20260619 02:37:01.699079 20779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:01.699434 20778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5948d44be97140d1a2529fa4f84e7dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948d44be97140d1a2529fa4f84e7dc9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42857 } } }
I20260619 02:37:01.699543 20778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:01.699420 20785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:37:01.702996 20785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:37:01.703703 20785 catalog_manager.cc:1259] Loaded cluster ID: a62624cfc8cd4d8ba3f9f702b441fc4e
I20260619 02:37:01.703671 20773 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:37:01.704697 20785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:37:01.710623 20785 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:37:01.711491 20785 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Loaded TSK: 0
I20260619 02:37:01.712831 20785 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:37:01.808193 20775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:01.808678 20775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:01.808799 20775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:01.815217 20775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:01.815389 20775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:01.815503 20775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:37:01.824131 20775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.194.212:58455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:33045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=42305
--tserver_master_addrs=127.7.194.254:42857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:01.825642 20775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:01.827042 20775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:01.839740 20810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:01.839911 20811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:01.839741 20813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:01.840343 20775 server_base.cc:1034] running on GCE node
I20260619 02:37:01.840865 20775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:01.841735 20775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:01.842936 20775 hybrid_clock.cc:648] HybridClock initialized: now 1781836621842915 us; error 36 us; skew 500 ppm
Jun 19 02:37:01 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:01.846587 20775 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:37:01.847829 20775 webserver.cc:460] Webserver started at http://127.7.194.193:42305/ using document root <none> and password file <none>
I20260619 02:37:01.848628 20775 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:01.848798 20775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:37:01 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:01.873340 20775 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 02:37:01.876217 20821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:01.878086 20775 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 02:37:01.878245 20775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "1bdfb0eb440c46bdb913508abfea9e4b"
format_stamp: "Formatted at 2026-06-19 02:37:00 on dist-test-slave-9tkz"
server_key: "028666708506e7891d8921168abc3aae"
server_key_iv: "fdb9f0dadd603123a3887a83fba2f670"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:01.878978 20775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:01.899204 20775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:01.903694 20775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:01.903913 20775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:01.904641 20775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:01.905771 20775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:01.905840 20775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:01.905896 20775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:01.905927 20775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:01.917670 20775 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:33045
I20260619 02:37:01.917937 20934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:33045 every 8 connection(s)
I20260619 02:37:01.918910 20775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:37:01.920411 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 20775
I20260619 02:37:01.920926 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:58455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:38389
--local_ip_for_outbound_sockets=127.7.194.194
--tserver_master_addrs=127.7.194.254:42857
--webserver_port=42927
--webserver_interface=127.7.194.194
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.194.212:58455/kms with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:37:01 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:01.957373 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:01.923329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:59571 (local address 127.7.194.254:42857)
0619 02:37:01.929884 (+ 6555us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:01.929911 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:01.929970 (+ 59us) server_negotiation.cc:407] Connection header received
0619 02:37:01.930338 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:01.930361 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:01.930880 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:01.941896 (+ 11016us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:01.943461 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:01.945166 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:01.947037 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:01.947858 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:01.948053 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:01.951196 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:01.951204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:01.951227 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:01.951265 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:01.953438 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:01.954142 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:01.954149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:01.954155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:01.954311 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:01.954731 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:01.954735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:01.954737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:01.955318 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:01.955463 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:01.956081 (+ 618us) server_negotiation.cc:299] Negotiation successful
0619 02:37:01.956408 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6336,"thread_start_us":119,"threads_started":1}
I20260619 02:37:01.959069 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:01.921866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:42857 (local address 127.7.194.193:59571)
0619 02:37:01.923442 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:01.923499 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:01.924422 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:01.941997 (+ 17575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:01.942007 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:01.942439 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:01.943179 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:01.943201 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:01.945350 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:01.945354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:01.946859 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:01.946873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:01.948070 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:01.948076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:01.948326 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:01.949066 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:01.949091 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:01.950954 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:01.953609 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:01.953616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:01.953630 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:01.953967 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:01.954422 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:01.954425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:01.954427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:01.954595 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:01.955449 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:01.955455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:01.955730 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 02:37:01.957879 (+ 2149us) client_negotiation.cc:240] Negotiation successful
0619 02:37:01.958208 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"thread_start_us":122,"threads_started":1}
I20260619 02:37:01.961860 20935 heartbeater.cc:347] Connected to a master server at 127.7.194.254:42857
I20260619 02:37:01.962116 20935 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:01.962675 20935 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
W20260619 02:37:01.964913 20935 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:42857 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:37:01.965044 20935 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:01.965282 20935 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
I20260619 02:37:01.966193 20935 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:01.966364 20935 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
W20260619 02:37:01.967082 20935 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:37:02.068678 20938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:02.068996 20938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:02.069083 20938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:02.073531 20938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:02.073679 20938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:02.073776 20938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:37:02.079557 20938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.194.212:58455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:38389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=42927
--tserver_master_addrs=127.7.194.254:42857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:02.080860 20938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:02.082079 20938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:02.092880 20944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:02.093154 20938 server_base.cc:1034] running on GCE node
W20260619 02:37:02.094403 20945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:02.094481 20947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:02.094902 20938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:02.095643 20938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:02.096817 20938 hybrid_clock.cc:648] HybridClock initialized: now 1781836622096788 us; error 48 us; skew 500 ppm
Jun 19 02:37:02 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836622, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:02.099773 20938 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:37:02.100939 20938 webserver.cc:460] Webserver started at http://127.7.194.194:42927/ using document root <none> and password file <none>
I20260619 02:37:02.101514 20938 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:02.101642 20938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:37:02 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836622, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:02.123921 20938 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:37:02.127182 20955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:02.128998 20938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:37:02.129140 20938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "7485526f008349bbaa9e6933fc3035ce"
format_stamp: "Formatted at 2026-06-19 02:37:00 on dist-test-slave-9tkz"
server_key: "5bb9afd060495ff6845ceb852c539c34"
server_key_iv: "2dc29ddd79159c35b97085a26433f1ad"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:02.129967 20938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:02.148432 20938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:02.152482 20938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:02.152700 20938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:02.153412 20938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:02.154525 20938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:02.154588 20938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:02.154685 20938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:02.154728 20938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:02.165405 20938 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:38389
I20260619 02:37:02.166783 20938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:37:02.173825 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 20938
I20260619 02:37:02.174382 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.194.212:58455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:39215
--local_ip_for_outbound_sockets=127.7.194.195
--tserver_master_addrs=127.7.194.254:42857
--webserver_port=43061
--webserver_interface=127.7.194.195
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.194.212:58455/kms with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:02.174588 21068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:38389 every 8 connection(s)
Jun 19 02:37:02 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836622, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:02.196722 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:02.176867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:34403 (local address 127.7.194.254:42857)
0619 02:37:02.177027 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:02.177034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:02.182185 (+ 5151us) server_negotiation.cc:407] Connection header received
0619 02:37:02.183603 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:02.183607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:02.183684 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:02.183741 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:02.185841 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.186673 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.188537 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.188916 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.189000 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:02.192296 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:02.192300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:02.192303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:02.192332 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:02.194205 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:02.194858 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:02.194862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:02.194864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:02.194956 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:02.195346 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:02.195349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:02.195351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:02.195569 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:02.195664 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:02.196369 (+ 705us) server_negotiation.cc:299] Negotiation successful
0619 02:37:02.196526 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:37:02.198509 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:02.177524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:42857 (local address 127.7.194.194:34403)
0619 02:37:02.181961 (+ 4437us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:02.182037 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:02.183260 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:02.184215 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:02.184226 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:02.184774 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:02.185624 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:02.185673 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.186847 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.186851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:02.188395 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:02.188405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.189222 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.189226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:02.189441 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:02.190240 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:02.190262 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:02.192088 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:02.194341 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:02.194347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:02.194365 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:02.194731 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:02.195060 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:02.195063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:02.195065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:02.195239 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:02.195735 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:02.195742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:02.196080 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 02:37:02.196915 (+ 835us) client_negotiation.cc:240] Negotiation successful
0619 02:37:02.197290 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4119,"thread_start_us":158,"threads_started":1}
I20260619 02:37:02.200045 21069 heartbeater.cc:347] Connected to a master server at 127.7.194.254:42857
I20260619 02:37:02.200345 21069 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:02.200984 21069 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
W20260619 02:37:02.202638 21069 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:42857 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:37:02.202768 21069 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:02.202965 21069 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
I20260619 02:37:02.203809 21069 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:02.203994 21069 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
W20260619 02:37:02.204746 21069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:37:02.337898 21071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:02.338234 21071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:02.338317 21071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:02.342741 21071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:02.342837 21071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:02.342950 21071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:37:02.349255 21071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.194.212:58455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:39215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=43061
--tserver_master_addrs=127.7.194.254:42857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:02.350493 21071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:02.351667 21071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:02.364246 21078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:02.364758 21079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:02.364246 21081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:02.366211 21071 server_base.cc:1034] running on GCE node
I20260619 02:37:02.366844 21071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:02.367815 21071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:02.371171 21071 hybrid_clock.cc:648] HybridClock initialized: now 1781836622371113 us; error 71 us; skew 500 ppm
Jun 19 02:37:02 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836622, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:02.375322 21071 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:37:02.376595 21071 webserver.cc:460] Webserver started at http://127.7.194.195:43061/ using document root <none> and password file <none>
I20260619 02:37:02.377375 21071 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:02.377489 21071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:37:02 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836622, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:02.403048 21071 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:37:02.406551 21089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:02.408725 21071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:37:02.408870 21071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "48255ac8f863483c85c8efdd4c09e1db"
format_stamp: "Formatted at 2026-06-19 02:37:01 on dist-test-slave-9tkz"
server_key: "16c15b066b5adfbb86e0cf47472f29b9"
server_key_iv: "bcd40c61d146ecd2aaeaee519ee1a2b6"
server_key_version: "kuduclusterkey@0"
I20260619 02:37:02.409667 21071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:02.425818 21071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:02.429258 21071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:02.429476 21071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:02.430130 21071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:02.431245 21071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:02.431319 21071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:02.431391 21071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:02.431424 21071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:02.443794 21071 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:39215
I20260619 02:37:02.443933 21202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:39215 every 8 connection(s)
I20260619 02:37:02.445137 21071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:37:02.446897 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 21071
Jun 19 02:37:02 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836622, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:02.459381 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:02.448400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.7.194.254:42857)
0619 02:37:02.449206 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:02.449209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:02.449217 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:37:02.449284 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:02.449290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:02.449345 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:02.449437 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:02.450613 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.451454 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.452802 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.453048 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.453168 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:02.454862 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:02.454865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:02.454867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:02.454893 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:02.456741 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:02.457274 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:02.457279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:02.457284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:02.457378 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:02.457771 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:02.457775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:02.457777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:02.458002 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:02.458205 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:02.458957 (+ 752us) server_negotiation.cc:299] Negotiation successful
0619 02:37:02.459235 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":77,"threads_started":1}
I20260619 02:37:02.462589 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:02.448133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:38181 (local address 127.7.194.254:42857)
0619 02:37:02.449022 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:02.449026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:02.449040 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:37:02.450009 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:02.450013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:02.450069 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:02.450139 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:02.451856 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.452647 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.454478 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.454826 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.454913 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:02.457982 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:02.457986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:02.457989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:02.458016 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:02.459741 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:02.460540 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:02.460543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:02.460545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:02.460634 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:02.461218 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:02.461222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:02.461224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:02.461438 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:02.461569 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:02.462285 (+ 716us) server_negotiation.cc:299] Negotiation successful
0619 02:37:02.462414 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812}
I20260619 02:37:02.463832 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:02.447973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:42857 (local address 127.7.194.195:38181)
0619 02:37:02.448558 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:02.448621 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:02.449751 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:02.450576 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:02.450586 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:02.450944 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:02.451667 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:02.451691 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.452815 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.452818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:02.454329 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:02.454341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:02.455165 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:02.455171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:02.455379 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:02.455957 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:02.455981 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:02.457773 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:02.459997 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:02.460004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:02.460024 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:02.460412 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:02.460752 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:02.460755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:02.460757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:02.461081 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:02.461587 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:02.461594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:02.461962 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 02:37:02.462202 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 02:37:02.462577 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":153,"threads_started":1}
I20260619 02:37:02.465188 21203 heartbeater.cc:347] Connected to a master server at 127.7.194.254:42857
I20260619 02:37:02.465433 21203 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:02.466039 21203 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
W20260619 02:37:02.467852 21203 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:42857 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:37:02.467970 21203 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:02.468123 21203 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
I20260619 02:37:02.468956 21203 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:02.469149 21203 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
W20260619 02:37:02.469959 21203 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 02:37:02 dist-test-slave-9tkz krb5kdc[19741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781836622, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:02.967340 20935 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:02.967707 20935 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
I20260619 02:37:02.968905 20737 ts_manager.cc:194] Registered new tserver with Master: 1bdfb0eb440c46bdb913508abfea9e4b (127.7.194.193:33045)
I20260619 02:37:02.970634 20737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:59571
I20260619 02:37:03.205011 21069 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:03.205345 21069 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
I20260619 02:37:03.206243 20737 ts_manager.cc:194] Registered new tserver with Master: 7485526f008349bbaa9e6933fc3035ce (127.7.194.194:38389)
I20260619 02:37:03.207082 20737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:34403
Jun 19 02:37:03 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836622, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:03.470191 21203 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:03.470498 21203 heartbeater.cc:511] Master 127.7.194.254:42857 requested a full tablet report, sending...
I20260619 02:37:03.471240 20737 ts_manager.cc:194] Registered new tserver with Master: 48255ac8f863483c85c8efdd4c09e1db (127.7.194.195:39215)
I20260619 02:37:03.471966 20737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:38181
I20260619 02:37:03.480928 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:37:03.498797 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.485126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.7.194.254:42857)
0619 02:37:03.485382 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:03.485386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:03.485939 (+ 553us) server_negotiation.cc:407] Connection header received
0619 02:37:03.486076 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:03.486080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:03.486149 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:03.486250 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:03.487140 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.487986 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.489216 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.489621 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.489727 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.491795 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:03.491799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:03.491802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:03.491830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:03.493770 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:03.494314 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:03.494318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:03.494320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:03.494450 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:03.494790 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:03.494794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:03.494796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:03.495006 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:03.495178 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:03.498407 (+ 3229us) server_negotiation.cc:299] Negotiation successful
0619 02:37:03.498589 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260619 02:37:03.503628 20737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:37:03.508842 20737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 19 02:37:03 dist-test-slave-9tkz krb5kdc[19741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836622, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for HTTP/127.7.194.212@KRBTEST.COM
I20260619 02:37:03.534071 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.523135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:38389 (local address 127.0.0.1:53768)
0619 02:37:03.527167 (+ 4032us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:03.527226 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:03.527363 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:03.528483 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:03.528490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:03.528520 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:03.528678 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.528686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.530629 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.530633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.532127 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.532138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.533476 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.533479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.533625 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.533666 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:37:03.533775 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 02:37:03.533901 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3826,"thread_start_us":116,"threads_started":1}
I20260619 02:37:03.534778 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.523851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.7.194.194:38389)
0619 02:37:03.527936 (+ 4085us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:03.527945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:03.527973 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:37:03.528053 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:03.528059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:03.528293 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:03.528448 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:03.528834 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.530483 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.532277 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.533157 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.533262 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.533414 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:03.534517 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0619 02:37:03.534636 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3978,"thread_start_us":68,"threads_started":1}
I20260619 02:37:03.538672 21002 tablet_service.cc:1467] Processing CreateTablet for tablet e7f785b760d0441594cbba4fdd13f693 (DEFAULT_TABLE table=test-table [id=88453427c67d460ebf4d297fb8a8e015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:03.540340 21002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f785b760d0441594cbba4fdd13f693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:03.541721 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.524682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:39215 (local address 127.0.0.1:39896)
0619 02:37:03.528877 (+ 4195us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:03.528891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:03.528978 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:03.531338 (+ 2360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:03.531341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:03.531352 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:03.531409 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.531414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.533567 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.533571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.537469 (+ 3898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.537481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.541285 (+ 3804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.541289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.541379 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.541403 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:37:03.541473 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:37:03.541573 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4137,"thread_start_us":80,"threads_started":1}
I20260619 02:37:03.542300 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.525080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.7.194.195:39215)
0619 02:37:03.527746 (+ 2666us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:03.527754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:03.529067 (+ 1313us) server_negotiation.cc:407] Connection header received
0619 02:37:03.529160 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:03.529165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:03.529361 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:03.530150 (+ 789us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:03.531517 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.533411 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.540141 (+ 6730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.540956 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.541061 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.541222 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:03.542040 (+ 818us) server_negotiation.cc:299] Negotiation successful
0619 02:37:03.542161 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2555,"thread_start_us":84,"threads_started":1}
I20260619 02:37:03.545156 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.525955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:33045 (local address 127.0.0.1:50996)
0619 02:37:03.530404 (+ 4449us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:03.530417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:03.530496 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:03.531137 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:03.531141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:03.531157 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:03.531242 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.531247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.537998 (+ 6751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.538002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.539971 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.539981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.544573 (+ 4592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.544579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.544702 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.544737 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:37:03.544825 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:37:03.544933 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4389,"thread_start_us":111,"threads_started":1}
I20260619 02:37:03.547027 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.526397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.7.194.193:33045)
0619 02:37:03.528743 (+ 2346us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:03.528749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:03.530589 (+ 1840us) server_negotiation.cc:407] Connection header received
0619 02:37:03.530681 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:03.530686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:03.530868 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:03.530992 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:03.536151 (+ 5159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.537874 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.543363 (+ 5489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.544236 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.544349 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.544497 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:03.546765 (+ 2268us) server_negotiation.cc:299] Negotiation successful
0619 02:37:03.546884 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2247,"thread_start_us":94,"threads_started":1}
I20260619 02:37:03.548693 21118 tablet_service.cc:1467] Processing CreateTablet for tablet e7f785b760d0441594cbba4fdd13f693 (DEFAULT_TABLE table=test-table [id=88453427c67d460ebf4d297fb8a8e015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:03.549970 21118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f785b760d0441594cbba4fdd13f693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:03.550758 20869 tablet_service.cc:1467] Processing CreateTablet for tablet e7f785b760d0441594cbba4fdd13f693 (DEFAULT_TABLE table=test-table [id=88453427c67d460ebf4d297fb8a8e015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:03.551748 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f785b760d0441594cbba4fdd13f693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:03.561637 21239 tablet_bootstrap.cc:492] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b: Bootstrap starting.
I20260619 02:37:03.564355 21239 tablet_bootstrap.cc:654] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:03.565811 21239 log.cc:824] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:03.565920 21240 tablet_bootstrap.cc:492] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce: Bootstrap starting.
I20260619 02:37:03.568205 21239 tablet_bootstrap.cc:492] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b: No bootstrap required, opened a new log
I20260619 02:37:03.568367 21240 tablet_bootstrap.cc:654] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:03.568410 21239 ts_tablet_manager.cc:1397] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 02:37:03.569235 21240 log.cc:824] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:03.570449 21238 tablet_bootstrap.cc:492] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: Bootstrap starting.
I20260619 02:37:03.572705 21239 raft_consensus.cc:348] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } }
I20260619 02:37:03.572983 21239 raft_consensus.cc:374] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:03.573032 21239 raft_consensus.cc:729] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdfb0eb440c46bdb913508abfea9e4b, State: Initialized, Role: FOLLOWER
I20260619 02:37:03.574047 21239 consensus_queue.cc:260] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } }
I20260619 02:37:03.574622 21238 tablet_bootstrap.cc:654] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:03.575402 21239 ts_tablet_manager.cc:1428] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 02:37:03.575819 20935 heartbeater.cc:503] Master 127.7.194.254:42857 was elected leader, sending a full tablet report...
I20260619 02:37:03.580071 21238 log.cc:824] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:03.586181 21240 tablet_bootstrap.cc:492] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce: No bootstrap required, opened a new log
I20260619 02:37:03.586432 21240 ts_tablet_manager.cc:1397] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.005s
I20260619 02:37:03.588505 21238 tablet_bootstrap.cc:492] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: No bootstrap required, opened a new log
I20260619 02:37:03.588729 21238 ts_tablet_manager.cc:1397] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260619 02:37:03.591709 21240 raft_consensus.cc:348] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } }
I20260619 02:37:03.592154 21240 raft_consensus.cc:374] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:03.592268 21240 raft_consensus.cc:729] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7485526f008349bbaa9e6933fc3035ce, State: Initialized, Role: FOLLOWER
I20260619 02:37:03.592931 21240 consensus_queue.cc:260] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } }
I20260619 02:37:03.592868 21238 raft_consensus.cc:348] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } }
I20260619 02:37:03.593160 21238 raft_consensus.cc:374] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:03.593226 21238 raft_consensus.cc:729] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48255ac8f863483c85c8efdd4c09e1db, State: Initialized, Role: FOLLOWER
I20260619 02:37:03.593750 21238 consensus_queue.cc:260] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } }
I20260619 02:37:03.594602 21240 ts_tablet_manager.cc:1428] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260619 02:37:03.594806 21238 ts_tablet_manager.cc:1428] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 02:37:03.595075 21069 heartbeater.cc:503] Master 127.7.194.254:42857 was elected leader, sending a full tablet report...
I20260619 02:37:03.595177 21203 heartbeater.cc:503] Master 127.7.194.254:42857 was elected leader, sending a full tablet report...
W20260619 02:37:03.668963 21070 tablet.cc:2369] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:37:03.676410 20936 tablet.cc:2369] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:37:03.697127 21204 tablet.cc:2369] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:37:03.700156 21243 raft_consensus.cc:482] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:37:03.700443 21243 raft_consensus.cc:504] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } }
I20260619 02:37:03.701810 21243 leader_election.cc:288] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7485526f008349bbaa9e6933fc3035ce (127.7.194.194:38389), 48255ac8f863483c85c8efdd4c09e1db (127.7.194.195:39215)
I20260619 02:37:03.707733 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.702302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:60571 (local address 127.7.194.195:39215)
0619 02:37:03.702439 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:03.702444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:03.702727 (+ 283us) server_negotiation.cc:407] Connection header received
0619 02:37:03.702967 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:03.702971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:03.703029 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:03.703083 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:03.703564 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.704324 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.706405 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.707014 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.707104 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.707156 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:03.707488 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 02:37:03.707575 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:37:03.708447 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.702228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:39215 (local address 127.7.194.193:60571)
0619 02:37:03.702616 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:03.702633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:03.702822 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:03.703242 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:03.703245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:03.703265 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:03.703417 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.703424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.704481 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.704485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.706270 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.706279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.707250 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.707253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.707347 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.707368 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:37:03.708192 (+ 824us) client_negotiation.cc:240] Negotiation successful
0619 02:37:03.708274 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":3,"thread_start_us":124,"threads_started":1}
I20260619 02:37:03.709295 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f785b760d0441594cbba4fdd13f693" candidate_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48255ac8f863483c85c8efdd4c09e1db" is_pre_election: true
I20260619 02:37:03.709685 21149 raft_consensus.cc:2454] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdfb0eb440c46bdb913508abfea9e4b in term 0.
I20260619 02:37:03.710237 20823 leader_election.cc:302] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdfb0eb440c46bdb913508abfea9e4b, 48255ac8f863483c85c8efdd4c09e1db; no voters:
I20260619 02:37:03.710567 21243 raft_consensus.cc:2781] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:37:03.710644 21243 raft_consensus.cc:482] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:37:03.710692 21243 raft_consensus.cc:3037] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:03.712076 21243 raft_consensus.cc:504] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } }
I20260619 02:37:03.712486 21243 leader_election.cc:288] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [CANDIDATE]: Term 1 election: Requested vote from peers 7485526f008349bbaa9e6933fc3035ce (127.7.194.194:38389), 48255ac8f863483c85c8efdd4c09e1db (127.7.194.195:39215)
I20260619 02:37:03.713155 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f785b760d0441594cbba4fdd13f693" candidate_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48255ac8f863483c85c8efdd4c09e1db"
I20260619 02:37:03.713289 21149 raft_consensus.cc:3037] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:03.713963 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.703111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:47215 (local address 127.7.194.194:38389)
0619 02:37:03.703250 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:03.703255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:03.705341 (+ 2086us) server_negotiation.cc:407] Connection header received
0619 02:37:03.705520 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:03.705524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:03.705596 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:03.705640 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:03.706666 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.707370 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.711598 (+ 4228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.712224 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.712321 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.712373 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:03.713753 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0619 02:37:03.713841 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:37:03.714744 21149 raft_consensus.cc:2454] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdfb0eb440c46bdb913508abfea9e4b in term 1.
I20260619 02:37:03.715240 20823 leader_election.cc:302] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdfb0eb440c46bdb913508abfea9e4b, 48255ac8f863483c85c8efdd4c09e1db; no voters:
I20260619 02:37:03.715461 21243 raft_consensus.cc:2781] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:37:03.715687 21243 raft_consensus.cc:686] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [term 1 LEADER]: Becoming Leader. State: Replica: 1bdfb0eb440c46bdb913508abfea9e4b, State: Running, Role: LEADER
I20260619 02:37:03.716089 21243 consensus_queue.cc:237] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } }
I20260619 02:37:03.717803 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.702707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:38389 (local address 127.7.194.193:47215)
0619 02:37:03.705252 (+ 2545us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:03.705267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:03.705408 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:03.705770 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:03.705773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:03.705789 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:03.706531 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.706539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.707499 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.707501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.711440 (+ 3939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:03.711451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.713431 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.713434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:03.713584 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.713621 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 02:37:03.715659 (+ 2038us) client_negotiation.cc:240] Negotiation successful
0619 02:37:03.717638 (+ 1979us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2475,"mutex_wait_us":216,"thread_start_us":104,"threads_started":1}
I20260619 02:37:03.719075 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f785b760d0441594cbba4fdd13f693" candidate_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7485526f008349bbaa9e6933fc3035ce"
I20260619 02:37:03.719107 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f785b760d0441594cbba4fdd13f693" candidate_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7485526f008349bbaa9e6933fc3035ce" is_pre_election: true
I20260619 02:37:03.719424 21023 raft_consensus.cc:2454] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdfb0eb440c46bdb913508abfea9e4b in term 0.
I20260619 02:37:03.722229 20735 catalog_manager.cc:5495] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bdfb0eb440c46bdb913508abfea9e4b (127.7.194.193). New cstate: current_term: 1 leader_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bdfb0eb440c46bdb913508abfea9e4b" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33045 } health_report { overall_health: HEALTHY } } }
I20260619 02:37:03.782366 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.777075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.7.194.193:33045)
0619 02:37:03.777209 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:03.777214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:03.777230 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:37:03.777440 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:03.777444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:03.777516 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:03.777667 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:37:03.777942 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.778776 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.780071 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:03.780420 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:03.780537 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:03.780996 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:37:03.781629 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:37:03.781768 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:03.782071 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:37:03.782193 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:37:03.788480 21022 raft_consensus.cc:3037] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:03.789222 21149 raft_consensus.cc:1261] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [term 1 FOLLOWER]: Refusing update from remote peer 1bdfb0eb440c46bdb913508abfea9e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:03.790024 21250 consensus_queue.cc:1035] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "48255ac8f863483c85c8efdd4c09e1db" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:03.790133 21022 raft_consensus.cc:1261] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce [term 1 FOLLOWER]: Refusing update from remote peer 1bdfb0eb440c46bdb913508abfea9e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:03.790788 21250 consensus_queue.cc:1035] T e7f785b760d0441594cbba4fdd13f693 P 1bdfb0eb440c46bdb913508abfea9e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7485526f008349bbaa9e6933fc3035ce" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:03.797837 21253 mvcc.cc:205] Tried to move back new op lower bound from 7298402811030532096 to 7298402810745090048. Current Snapshot: MvccSnapshot[applied={T|T < 7298402811030532096}]
I20260619 02:37:03.800221 21255 mvcc.cc:205] Tried to move back new op lower bound from 7298402811030532096 to 7298402810745090048. Current Snapshot: MvccSnapshot[applied={T|T < 7298402811030532096}]
I20260619 02:37:03.809741 20737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:03.809937 20737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:03.812727 20737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5948d44be97140d1a2529fa4f84e7dc9: Sending DeleteTablet for 3 replicas of tablet e7f785b760d0441594cbba4fdd13f693
I20260619 02:37:03.814071 21002 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f785b760d0441594cbba4fdd13f693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:03 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:53768
I20260619 02:37:03.814630 20869 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f785b760d0441594cbba4fdd13f693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:03 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:50996
I20260619 02:37:03.815439 21118 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f785b760d0441594cbba4fdd13f693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:03 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:39896
I20260619 02:37:03.815963 21264 tablet_replica.cc:330] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: stopping tablet replica
I20260619 02:37:03.816323 21264 raft_consensus.cc:2229] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:03.816619 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 20775
I20260619 02:37:03.817286 21264 raft_consensus.cc:2258] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:03.820163 21254 mvcc.cc:205] Tried to move back new op lower bound from 7298402811030532096 to 7298402810745090048. Current Snapshot: MvccSnapshot[applied={T|T < 7298402811030532096}]
I20260619 02:37:03.820441 21261 tablet_replica.cc:330] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce: stopping tablet replica
I20260619 02:37:03.820768 21261 raft_consensus.cc:2229] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:03.821029 21261 raft_consensus.cc:2258] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:03.825654 21264 ts_tablet_manager.cc:1905] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:37:03.826367 21261 ts_tablet_manager.cc:1905] T e7f785b760d0441594cbba4fdd13f693 P 7485526f008349bbaa9e6933fc3035ce: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:37:03.827677 20724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:33045 (error 108)
I20260619 02:37:03.828567 21264 ts_tablet_manager.cc:1918] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:03.828655 21264 log.cc:1196] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/e7f785b760d0441594cbba4fdd13f693
I20260619 02:37:03.828869 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 20938
I20260619 02:37:03.829063 21264 ts_tablet_manager.cc:1939] T e7f785b760d0441594cbba4fdd13f693 P 48255ac8f863483c85c8efdd4c09e1db: Deleting consensus metadata
I20260619 02:37:03.830466 20722 catalog_manager.cc:4841] TS 48255ac8f863483c85c8efdd4c09e1db (127.7.194.195:39215): tablet e7f785b760d0441594cbba4fdd13f693 (table test-table [id=88453427c67d460ebf4d297fb8a8e015]) successfully deleted
I20260619 02:37:03.833276 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.832961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:33045 (local address 127.0.0.1:51006)
0619 02:37:03.833115 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:03.833186 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 02:37:03.833441 20724 catalog_manager.cc:4584] TS 1bdfb0eb440c46bdb913508abfea9e4b (127.7.194.193:33045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7f785b760d0441594cbba4fdd13f693: Network error: Client connection negotiation failed: client connection to 127.7.194.193:33045: connect: Connection refused (error 111)
I20260619 02:37:03.838279 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 21071
I20260619 02:37:03.838652 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:03.838345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:38389 (local address 127.0.0.1:53782)
0619 02:37:03.838514 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:03.838566 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.194:38389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 02:37:03.845171 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 20704
I20260619 02:37:03.853443 7947 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:37:04.188310 7947 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:37:04.188460 7947 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:37:04.533778 7947 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:37:04.534 UTC [19771] LOG: received smart shutdown request
2026-06-19 02:37:04.537 UTC [19771] LOG: background worker "logical replication launcher" (PID 19780) exited with exit code 1
2026-06-19 02:37:04.538 UTC [19775] LOG: shutting down
2026-06-19 02:37:04.706 UTC [19771] LOG: database system is shut down
2026-06-19T02:37:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49529 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:37:04 dist-test-slave-9tkz krb5kdc[21281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:37:04 dist-test-slave-9tkz krb5kdc[21281](info): set up 2 sockets
Jun 19 02:37:04 dist-test-slave-9tkz krb5kdc[21281](info): commencing operation
krb5kdc: starting...
W20260619 02:37:05.866227 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:37:05 dist-test-slave-9tkz krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:37:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:37:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:37:05.991861 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:37137
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:37137,127.7.194.253:45965,127.7.194.252:38797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:06.105342 21294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:06.105712 21294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:06.105795 21294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:06.109879 21294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:06.109989 21294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:06.110025 21294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:06.110065 21294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:06.110092 21294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:06.115628 21294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:37137,127.7.194.253:45965,127.7.194.252:38797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:37137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:06.116825 21294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:06.117975 21294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:37:06.126838 21294 server_base.cc:1034] running on GCE node
W20260619 02:37:06.126812 21299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:06.126801 21302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:06.126801 21300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:06.127621 21294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:06.128783 21294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:06.129995 21294 hybrid_clock.cc:648] HybridClock initialized: now 1781836626129979 us; error 36 us; skew 500 ppm
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:06.132869 21294 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:37:06.133921 21294 webserver.cc:460] Webserver started at http://127.7.194.254:39811/ using document root <none> and password file <none>
I20260619 02:37:06.134490 21294 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:06.134591 21294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:06.134826 21294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:06.136947 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "b9b9e58eff874edeb837df99b7714e25"
format_stamp: "Formatted at 2026-06-19 02:37:06 on dist-test-slave-9tkz"
server_key: "07d70a6ae0d1b139703206027b8698da"
server_key_iv: "4e1c3e229d544c4b68af80ae11631e7e"
server_key_version: "encryptionkey@0"
I20260619 02:37:06.137533 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "b9b9e58eff874edeb837df99b7714e25"
format_stamp: "Formatted at 2026-06-19 02:37:06 on dist-test-slave-9tkz"
server_key: "07d70a6ae0d1b139703206027b8698da"
server_key_iv: "4e1c3e229d544c4b68af80ae11631e7e"
server_key_version: "encryptionkey@0"
I20260619 02:37:06.141475 21294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:37:06.144057 21309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:06.145444 21294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:37:06.145675 21294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "b9b9e58eff874edeb837df99b7714e25"
format_stamp: "Formatted at 2026-06-19 02:37:06 on dist-test-slave-9tkz"
server_key: "07d70a6ae0d1b139703206027b8698da"
server_key_iv: "4e1c3e229d544c4b68af80ae11631e7e"
server_key_version: "encryptionkey@0"
I20260619 02:37:06.145807 21294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:06.175580 21294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:06.179531 21294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:06.179782 21294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:06.187866 21294 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:37137
I20260619 02:37:06.187879 21361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:37137 every 8 connection(s)
I20260619 02:37:06.188941 21294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:37:06.191939 21362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:06.195822 21362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:37:06.198838 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 21294
I20260619 02:37:06.198962 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:37:06.199265 7947 external_mini_cluster.cc:1428] Setting key 2dfd2040cafb9b135a182c2851acb2f0
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.253@KRBTEST.COM" created.
I20260619 02:37:06.213442 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.198318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.7.194.254:37137)
0619 02:37:06.198910 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.198928 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.199122 (+ 194us) server_negotiation.cc:407] Connection header received
0619 02:37:06.200636 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.200641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.200882 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.201100 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.201977 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.203427 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.204546 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.205235 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.205439 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.208286 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.208295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.208314 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.208346 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.210113 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.210667 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.210676 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.210681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.210824 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.211189 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.211191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.211192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.211581 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.211684 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.212265 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.212587 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":166,"server-negotiator.queue_time_us":473,"spinlock_wait_cycles":28032,"thread_start_us":170,"threads_started":1}
I20260619 02:37:06.213440 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.198318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:37137 (local address 127.0.0.1:38992)
0619 02:37:06.198913 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.198974 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:06.200290 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:06.201124 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:06.201129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:06.201459 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:06.201813 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.201828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.203562 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.203564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.204403 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.204412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.205394 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.205397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.205489 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.206285 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:06.206302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:06.208055 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:06.210236 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.210247 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.210261 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.210562 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.210919 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.210922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.210925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.211095 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.211698 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:06.211709 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:06.211894 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 02:37:06.212160 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 02:37:06.212485 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":40,"thread_start_us":155,"threads_started":1}
I20260619 02:37:06.216508 21362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:37:06.217397 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.217071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:45965 (local address 127.0.0.1:44762)
0619 02:37:06.217236 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.217319 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 02:37:06.217638 21312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
I20260619 02:37:06.218518 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.218256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:45965 (local address 127.0.0.1:44778)
0619 02:37:06.218388 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.218442 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 02:37:06.218667 21362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.253:45965: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab.
Entry for principal kudu/127.7.194.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab.
I20260619 02:37:06.226653 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.253:45965
--webserver_interface=127.7.194.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab
--principal=kudu/127.7.194.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:37137,127.7.194.253:45965,127.7.194.252:38797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:06.267983 21362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } attempt: 1
I20260619 02:37:06.269121 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.268746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:45965 (local address 127.0.0.1:44784)
0619 02:37:06.268931 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.269000 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 02:37:06.270040 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.269828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:45965 (local address 127.0.0.1:44790)
0619 02:37:06.269938 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.269985 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 02:37:06.270221 21362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.253:45965: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
I20260619 02:37:06.338400 21362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } attempt: 2
I20260619 02:37:06.339308 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.338982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:45965 (local address 127.0.0.1:44798)
0619 02:37:06.339170 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.339232 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 02:37:06.340124 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.339863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:45965 (local address 127.0.0.1:44814)
0619 02:37:06.339994 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.340059 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 02:37:06.340287 21362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.253:45965: Network error: Client connection negotiation failed: client connection to 127.7.194.253:45965: connect: Connection refused (error 111)
W20260619 02:37:06.340734 21369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:06.340982 21369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:06.341024 21369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:06.345108 21369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:06.345165 21369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:06.345185 21369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:06.345202 21369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:06.345217 21369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:06.350786 21369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:37137,127.7.194.253:45965,127.7.194.252:38797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab
--principal=kudu/127.7.194.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.253:45965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.194.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:06.351939 21369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:06.352993 21369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:06.361527 21377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:06.361557 21375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:06.361864 21369 server_base.cc:1034] running on GCE node
W20260619 02:37:06.361549 21374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:06.362524 21369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:06.363688 21369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:06.364888 21369 hybrid_clock.cc:648] HybridClock initialized: now 1781836626364873 us; error 33 us; skew 500 ppm
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:06.372531 21369 init.cc:377] Logged in from keytab as kudu/127.7.194.253@KRBTEST.COM (short username kudu)
I20260619 02:37:06.373514 21369 webserver.cc:460] Webserver started at http://127.7.194.253:44909/ using document root <none> and password file <none>
I20260619 02:37:06.374163 21369 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:06.374255 21369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:06.374466 21369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:06.376549 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/data/instance:
uuid: "0b8cb068fd6f4175b0003d9982e641cb"
format_stamp: "Formatted at 2026-06-19 02:37:06 on dist-test-slave-9tkz"
server_key: "a96d7da12042ab7d359b489ffde7667b"
server_key_iv: "a6ebfc8ce09c492df719326786f25bb7"
server_key_version: "encryptionkey@0"
I20260619 02:37:06.377135 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/wal/instance:
uuid: "0b8cb068fd6f4175b0003d9982e641cb"
format_stamp: "Formatted at 2026-06-19 02:37:06 on dist-test-slave-9tkz"
server_key: "a96d7da12042ab7d359b489ffde7667b"
server_key_iv: "a6ebfc8ce09c492df719326786f25bb7"
server_key_version: "encryptionkey@0"
I20260619 02:37:06.380914 21369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 02:37:06.383450 21384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:06.384703 21369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:37:06.384860 21369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/wal
uuid: "0b8cb068fd6f4175b0003d9982e641cb"
format_stamp: "Formatted at 2026-06-19 02:37:06 on dist-test-slave-9tkz"
server_key: "a96d7da12042ab7d359b489ffde7667b"
server_key_iv: "a6ebfc8ce09c492df719326786f25bb7"
server_key_version: "encryptionkey@0"
I20260619 02:37:06.384974 21369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:06.411207 21369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:06.415598 21369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:06.415817 21369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:06.424726 21369 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.253:45965
I20260619 02:37:06.424767 21436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.253:45965 every 8 connection(s)
I20260619 02:37:06.426086 21369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
I20260619 02:37:06.430224 21437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:06.431437 21362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } attempt: 3
I20260619 02:37:06.433624 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 21369
I20260619 02:37:06.433751 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-1/wal/instance
I20260619 02:37:06.434058 7947 external_mini_cluster.cc:1428] Setting key 8347578b0a6881571fb162b5d7cd4c51
I20260619 02:37:06.436668 21437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.252@KRBTEST.COM" created.
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:06.449724 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.432011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:45965 (local address 127.0.0.1:44828)
0619 02:37:06.432179 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.432198 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:06.432304 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:06.435403 (+ 3099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:06.435409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:06.435655 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:06.435848 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.435859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.437975 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.437982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.438972 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.438986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.441662 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.441666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.441824 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.442376 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:06.442399 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:06.444083 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:06.446601 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.446606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.446608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.446865 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.447331 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.447334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.447336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.447440 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.448227 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:06.448230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:06.448368 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 02:37:06.448492 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 02:37:06.449466 (+ 974us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:37:06.450181 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.432753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.7.194.253:45965)
0619 02:37:06.433158 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.433176 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.433223 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:37:06.434557 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.434595 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.435044 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.436091 (+ 1047us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.436480 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.437818 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.439141 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.439880 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.442166 (+ 2286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.444282 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.444288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.444302 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.444334 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.446480 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.447021 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.447030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.447040 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.447204 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.447543 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.447548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.447549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.448112 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.448253 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.448767 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.449010 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":138,"threads_started":1}
I20260619 02:37:06.451239 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.439056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:37137 (local address 127.0.0.1:38998)
0619 02:37:06.439455 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.439507 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:06.439640 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:06.439941 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:06.439950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:06.440497 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:06.440704 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.440715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.441926 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.441932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.443024 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.443037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.444282 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.444286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.444409 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.445187 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:06.445205 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:06.447322 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:06.449425 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.449434 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.449439 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.449756 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.450109 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.450112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.450114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.450203 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.450760 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:06.450764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:06.450862 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 02:37:06.450972 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:37:06.451112 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":153,"threads_started":1}
I20260619 02:37:06.451359 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.439093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.7.194.254:37137)
0619 02:37:06.439229 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.439235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.439601 (+ 366us) server_negotiation.cc:407] Connection header received
0619 02:37:06.439731 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.439734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.439793 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.439886 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.440851 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.441774 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.443161 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.444149 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.444242 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.447477 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.447484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.447490 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.447523 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.449300 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.449920 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.449923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.449925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.450007 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.450391 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.450394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.450395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.450610 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.450688 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.451076 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.451199 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:37:06.452261 21437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:37:06.452544 21362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:37:06.453198 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.452999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45236)
0619 02:37:06.453094 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.453148 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 02:37:06.453997 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.453721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45248)
0619 02:37:06.453889 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.453942 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260619 02:37:06.454187 21362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:38797: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
I20260619 02:37:06.463961 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.452815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:45965 (local address 127.0.0.1:44834)
0619 02:37:06.452936 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.452952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:06.453030 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:06.453905 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:06.453911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:06.454103 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:06.454236 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.454247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.454968 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.454974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.455877 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.455888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.456435 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.456440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.456539 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.457014 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:06.457034 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:06.460478 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:06.462533 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.462536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.462537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.462734 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.462985 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.462987 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.462989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.463059 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.463422 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:06.463424 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:06.463502 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 02:37:06.463580 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:37:06.463800 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 02:37:06.463984 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.452913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.7.194.253:45965)
0619 02:37:06.453686 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.453690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.453703 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:37:06.453735 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.453738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.453781 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.453863 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.454357 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.454859 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.456035 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.456307 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.456399 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.460726 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.460730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.460732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.460756 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.462417 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.462815 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.462817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.462818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.462888 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.463141 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.463144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.463145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.463343 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.463419 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.463659 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.463856 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":698}
I20260619 02:37:06.465174 21437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:37:06.466012 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.465729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45250)
0619 02:37:06.465859 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.465953 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab.
Entry for principal kudu/127.7.194.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab.
W20260619 02:37:06.466287 21388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
I20260619 02:37:06.466962 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.252:38797
--webserver_interface=127.7.194.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab
--principal=kudu/127.7.194.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:37137,127.7.194.253:45965,127.7.194.252:38797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:06.467185 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.466977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45252)
0619 02:37:06.467087 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.467130 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 02:37:06.467347 21437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:38797: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
I20260619 02:37:06.485394 21362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } attempt: 1
I20260619 02:37:06.486375 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.485988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45260)
0619 02:37:06.486183 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.486279 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:37:06.487195 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.486969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45276)
0619 02:37:06.487080 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.487133 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:37:06.487368 21362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:38797: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
I20260619 02:37:06.516642 21437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } attempt: 1
I20260619 02:37:06.517743 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.517320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45292)
0619 02:37:06.517501 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.517651 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 02:37:06.518501 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.518276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45298)
0619 02:37:06.518388 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.518441 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 02:37:06.518649 21437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:38797: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
I20260619 02:37:06.562541 21362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } attempt: 2
I20260619 02:37:06.563462 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.563143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45310)
0619 02:37:06.563309 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.563378 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 02:37:06.564237 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.564007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45320)
0619 02:37:06.564118 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.564174 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 02:37:06.564373 21362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:38797: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
W20260619 02:37:06.583618 21444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:06.584015 21444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:06.584071 21444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:37:06.586812 21437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } attempt: 2
W20260619 02:37:06.589776 21444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:06.589843 21444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:06.589869 21444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:06.589897 21444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:06.589921 21444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:06.592921 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.587456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45330)
0619 02:37:06.592696 (+ 5240us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.592786 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5098}
I20260619 02:37:06.593945 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.593627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45344)
0619 02:37:06.593812 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.593873 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260619 02:37:06.594132 21437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:38797: Network error: Client connection negotiation failed: client connection to 127.7.194.252:38797: connect: Connection refused (error 111)
I20260619 02:37:06.597766 21444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:37137,127.7.194.253:45965,127.7.194.252:38797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab
--principal=kudu/127.7.194.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.252:38797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.194.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:06.598870 21444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:06.599879 21444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:06.608395 21449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:06.608439 21450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:06.608395 21452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:06.608774 21444 server_base.cc:1034] running on GCE node
I20260619 02:37:06.609397 21444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:06.610649 21444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:06.611816 21444 hybrid_clock.cc:648] HybridClock initialized: now 1781836626611799 us; error 30 us; skew 500 ppm
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:06.614833 21444 init.cc:377] Logged in from keytab as kudu/127.7.194.252@KRBTEST.COM (short username kudu)
I20260619 02:37:06.615864 21444 webserver.cc:460] Webserver started at http://127.7.194.252:33393/ using document root <none> and password file <none>
I20260619 02:37:06.616447 21444 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:06.616539 21444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:06.616776 21444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:06.618921 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/data/instance:
uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd"
format_stamp: "Formatted at 2026-06-19 02:37:06 on dist-test-slave-9tkz"
server_key: "8f871a13c742cbc2be51c299ab90e07f"
server_key_iv: "366de3869aee9ae21b6e1764e690f5b3"
server_key_version: "encryptionkey@0"
I20260619 02:37:06.619504 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/wal/instance:
uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd"
format_stamp: "Formatted at 2026-06-19 02:37:06 on dist-test-slave-9tkz"
server_key: "8f871a13c742cbc2be51c299ab90e07f"
server_key_iv: "366de3869aee9ae21b6e1764e690f5b3"
server_key_version: "encryptionkey@0"
I20260619 02:37:06.623504 21444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:37:06.625957 21459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:06.627236 21444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:37:06.627386 21444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/wal
uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd"
format_stamp: "Formatted at 2026-06-19 02:37:06 on dist-test-slave-9tkz"
server_key: "8f871a13c742cbc2be51c299ab90e07f"
server_key_iv: "366de3869aee9ae21b6e1764e690f5b3"
server_key_version: "encryptionkey@0"
I20260619 02:37:06.627502 21444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:06.636803 21444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:06.639802 21444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:06.640012 21444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:06.647161 21444 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.252:38797
I20260619 02:37:06.647171 21511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.252:38797 every 8 connection(s)
I20260619 02:37:06.648128 21444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
I20260619 02:37:06.651028 21512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:06.653875 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 21444
I20260619 02:37:06.654067 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781836464269074-7947-0/minicluster-data/master-2/wal/instance
I20260619 02:37:06.654371 7947 external_mini_cluster.cc:1428] Setting key a5ad3039ed68e1e8947be8b381baca55
I20260619 02:37:06.655495 21512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:37:06.663548 21362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } attempt: 3
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
I20260619 02:37:06.669507 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.657428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.7.194.254:37137)
0619 02:37:06.657606 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.657612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.657989 (+ 377us) server_negotiation.cc:407] Connection header received
0619 02:37:06.658228 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.658232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.658295 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.658400 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.659251 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.660034 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.661436 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.661851 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.661965 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.665534 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.665538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.665540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.665592 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.667407 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.667971 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.667974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.667976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.668063 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.668420 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.668423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.668425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.668663 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.668760 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.669184 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.669303 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:37:06.670687 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.657381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:37137 (local address 127.0.0.1:39000)
0619 02:37:06.657837 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.657890 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:06.658091 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:06.658481 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:06.658486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:06.658929 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:06.659097 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.659109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.660147 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.660149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.661319 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.661332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.661992 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.661995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.662158 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.662948 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:06.662971 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:06.665355 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:06.667517 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.667525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.667537 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.667871 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.668164 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.668167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.668169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.668318 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.668757 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:06.668764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:06.668917 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 02:37:06.669747 (+ 830us) client_negotiation.cc:240] Negotiation successful
0619 02:37:06.669935 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":51,"thread_start_us":186,"threads_started":1}
I20260619 02:37:06.670692 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.655727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.7.194.252:38797)
0619 02:37:06.656222 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.656239 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.656277 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:37:06.657025 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.657048 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.657476 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.657875 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.658801 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.660238 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.661409 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.662090 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.662206 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.665433 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.665439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.665454 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.665477 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.667354 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.667853 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.667860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.667862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.667988 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.668303 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.668306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.668309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.668756 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.668950 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.669197 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.669500 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":332,"spinlock_wait_cycles":1920,"thread_start_us":194,"threads_started":1}
I20260619 02:37:06.672226 21512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:37:06.674199 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.664048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45360)
0619 02:37:06.664168 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.664184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:06.664285 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:06.664629 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:06.664632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:06.664810 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:06.664936 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.664944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.665852 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.665855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.666728 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.666739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.667378 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.667384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.667483 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.668139 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:06.668157 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:06.669716 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:06.672382 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.672385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.672387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.672619 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.673068 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.673071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.673073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.673203 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.673655 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:06.673661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:06.673768 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 02:37:06.673881 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 02:37:06.674025 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 02:37:06.674230 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.664089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.7.194.252:38797)
0619 02:37:06.664406 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.664413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.664423 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:37:06.664469 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.664471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.664511 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.664613 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.665077 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.665760 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.666857 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.667249 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.667366 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.669857 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.669861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.669863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.669885 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.672261 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.672744 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.672747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.672749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.672826 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.673318 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.673321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.673323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.673526 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.673643 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.673944 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.674052 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":118,"threads_started":1}
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
I20260619 02:37:06.677897 21362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25: Bootstrap starting.
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:06.681479 21362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:06.682310 21362 log.cc:824] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:06.682825 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.672831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.7.194.253:45965)
0619 02:37:06.672975 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.672979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.673020 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:37:06.673133 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.673138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.673196 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.673277 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.673795 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.674587 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.675629 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.675946 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.676021 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.678389 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.678393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.678395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.678423 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.680156 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.680613 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.680617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.680619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.680702 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.681325 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.681332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.681335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.681984 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.682071 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.682524 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.682662 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:37:06.683336 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.672740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:45965 (local address 127.0.0.1:44844)
0619 02:37:06.672953 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.672967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:06.673053 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:06.673393 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:06.673396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:06.673576 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:06.673676 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.673682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.674703 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.674708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.675481 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.675490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.676157 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.676159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.676215 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.676655 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:06.676667 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:06.678214 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:06.680323 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.680325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.680327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.680507 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.680818 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.680821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.680823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.680906 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.682148 (+ 1242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:06.682151 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:06.682267 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 02:37:06.682979 (+ 712us) client_negotiation.cc:240] Negotiation successful
0619 02:37:06.683128 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260619 02:37:06.684074 21512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:37:06.684506 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.673380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.7.194.254:37137)
0619 02:37:06.673478 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.673484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.673681 (+ 197us) server_negotiation.cc:407] Connection header received
0619 02:37:06.673843 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.673848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.673892 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.673937 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.675077 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.675915 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.676919 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.677281 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.677390 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.680725 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.680732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.680738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.680769 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.682500 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.683088 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.683094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.683096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.683174 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.683492 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.683495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.683497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.683722 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.683858 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.684220 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.684333 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:37:06.685067 21362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25: No bootstrap required, opened a new log
I20260619 02:37:06.685453 21437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } attempt: 3
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
I20260619 02:37:06.691135 21362 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } }
I20260619 02:37:06.691399 21362 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:06.691462 21362 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b9e58eff874edeb837df99b7714e25, State: Initialized, Role: FOLLOWER
Jun 19 02:37:06 dist-test-slave-9tkz krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
I20260619 02:37:06.692070 21362 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } }
I20260619 02:37:06.693692 21362 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:37:06.693837 21519 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.694026 21519 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:37:06.695555 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.684586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45376)
0619 02:37:06.684955 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.684966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:06.685066 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:06.685370 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:06.685372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:06.685529 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:06.685638 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.685646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.686597 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.686603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.687288 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.687297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.687905 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.687908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.687974 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.688905 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:06.688915 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:06.690957 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:06.692976 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.692978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.692980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.693288 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.694432 (+ 1144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.694438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.694442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.694548 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.695006 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:06.695009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:06.695113 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 02:37:06.695188 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:37:06.695331 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20260619 02:37:06.696094 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.686080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.7.194.252:38797)
0619 02:37:06.686164 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.686168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.686180 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:37:06.686210 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.686212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.686244 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.686317 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.686785 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.687383 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.688390 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.688710 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.688800 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.691187 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.691194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.691199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.691231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.694096 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.694570 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.694575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.694579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.694656 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.694970 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.694977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.694981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.695214 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.695312 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.695817 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.695930 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:37:06.696509 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.685815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:38797 (local address 127.0.0.1:45392)
0619 02:37:06.685932 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:06.685947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:06.686012 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:06.686334 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:06.686340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:06.686516 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:06.686642 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.686651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.687469 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.687472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.688284 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:06.688297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.688820 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.688827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:06.688918 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.689351 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:06.689365 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:06.691031 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:06.694217 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.694223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.694228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.694465 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.694775 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:06.694778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:06.694780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:06.694871 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:06.695348 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:06.695350 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:06.695443 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 02:37:06.695539 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:37:06.696333 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 02:37:06.696759 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.684810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.7.194.252:38797)
0619 02:37:06.684899 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.684903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.685029 (+ 126us) server_negotiation.cc:407] Connection header received
0619 02:37:06.685156 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.685161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.685216 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.685276 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.685757 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.686502 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.687400 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.687726 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.687817 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.691099 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.691103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.691105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.691128 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.692871 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.693426 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.693430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.693433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.694323 (+ 890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.694666 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.694670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.694673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.694893 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.695943 (+ 1050us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.696023 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.696412 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:37:06.696851 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.685937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.7.194.253:45965)
0619 02:37:06.686018 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.686022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.686035 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:37:06.686077 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.686083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.686115 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.686178 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.686967 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.687658 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.688908 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.689471 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.689570 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.692485 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.692493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.692497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.692525 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.694197 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.694785 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.694789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.694790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.695023 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.695322 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.695327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.695330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.695511 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.695712 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.696062 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.696726 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":548,"server-negotiator.queue_time_us":29}
I20260619 02:37:06.699800 21437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb: Bootstrap starting.
W20260619 02:37:06.700970 21530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:37:06.701054 21530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:37:06.698762 21294 master_runner.cc:186] Error getting master registration for 127.7.194.253:45965: OK, instance_id { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" instance_seqno: 1781836626422310 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:37:06.702713 21294 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:37:06.704010 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:37:06.704342 21512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd: Bootstrap starting.
I20260619 02:37:06.705653 21437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:06.706609 21437 log.cc:824] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:06.707015 21512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:06.707844 21512 log.cc:824] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:06.710167 21512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd: No bootstrap required, opened a new log
I20260619 02:37:06.714448 21437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb: No bootstrap required, opened a new log
I20260619 02:37:06.714424 21512 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } }
I20260619 02:37:06.714694 21512 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:06.714764 21512 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb7586d3b9749e2a1f2d0ee4ebc0bfd, State: Initialized, Role: FOLLOWER
I20260619 02:37:06.715304 21512 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } }
I20260619 02:37:06.716758 21512 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:37:06.716841 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.706204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.7.194.254:37137)
0619 02:37:06.706320 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.706324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.707430 (+ 1106us) server_negotiation.cc:407] Connection header received
0619 02:37:06.707625 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.707629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.707694 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.707750 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.708876 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.709747 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.710857 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.711203 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.711289 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.713135 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.713139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.713142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.713166 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.714924 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.715412 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.715416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.715417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.715505 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.715868 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.715872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.715873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.716082 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.716179 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.716535 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.716655 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:37:06.718746 21437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } }
I20260619 02:37:06.719002 21437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:06.719060 21437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8cb068fd6f4175b0003d9982e641cb, State: Initialized, Role: FOLLOWER
I20260619 02:37:06.719570 21437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } }
I20260619 02:37:06.721055 21437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:37:06.723790 21552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:37:06.723879 21552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:37:06.725708 21541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.725835 21541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:37:06.726060 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.726191 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:37:06.723430 21444 master_runner.cc:186] Error getting master registration for 127.7.194.253:45965: OK, instance_id { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" instance_seqno: 1781836626422310 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:37:06.727088 21444 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:37:06.728853 21369 master_runner.cc:263] No leader master found from master 0b8cb068fd6f4175b0003d9982e641cb
I20260619 02:37:06.728931 21369 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 02:37:06.729350 21565 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:37:06.729420 21565 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:37:06.733925 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.706860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.7.194.253:45965)
0619 02:37:06.706977 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.706983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.708134 (+ 1151us) server_negotiation.cc:407] Connection header received
0619 02:37:06.708333 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.708337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.708392 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.708451 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.709366 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.710108 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.723285 (+ 13177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.723649 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.723734 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.728862 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.728866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.728869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.728894 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.730828 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.732452 (+ 1624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.732456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.732458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.732548 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.732870 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.732874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.732876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.733097 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.733216 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.733625 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.733755 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:37:06.738098 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:06.707519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.7.194.252:38797)
0619 02:37:06.707628 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:06.707632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:06.711291 (+ 3659us) server_negotiation.cc:407] Connection header received
0619 02:37:06.711892 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:06.711896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:06.711946 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:06.712009 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:06.724709 (+ 12700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.725533 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.727842 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:06.728147 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:06.728233 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:06.734415 (+ 6182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:06.734420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:06.734422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:06.734449 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:06.736152 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.736662 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.736669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.736670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.736760 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:06.737060 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:06.737064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:06.737066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:06.737292 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:06.737419 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:06.737808 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 02:37:06.737936 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:37:06.809811 21519 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:37:06.810053 21519 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } }
I20260619 02:37:06.811244 21519 leader_election.cc:288] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b8cb068fd6f4175b0003d9982e641cb (127.7.194.253:45965), ecb7586d3b9749e2a1f2d0ee4ebc0bfd (127.7.194.252:38797)
I20260619 02:37:06.812127 21411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9b9e58eff874edeb837df99b7714e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8cb068fd6f4175b0003d9982e641cb" is_pre_election: true
I20260619 02:37:06.812475 21411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b9e58eff874edeb837df99b7714e25 in term 0.
I20260619 02:37:06.813781 21312 leader_election.cc:302] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8cb068fd6f4175b0003d9982e641cb, b9b9e58eff874edeb837df99b7714e25; no voters:
I20260619 02:37:06.814505 21486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9b9e58eff874edeb837df99b7714e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" is_pre_election: true
I20260619 02:37:06.814818 21486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b9e58eff874edeb837df99b7714e25 in term 0.
I20260619 02:37:06.815403 21519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:37:06.815505 21519 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:37:06.815558 21519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:06.816951 21519 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } }
I20260619 02:37:06.817349 21519 leader_election.cc:288] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [CANDIDATE]: Term 1 election: Requested vote from peers 0b8cb068fd6f4175b0003d9982e641cb (127.7.194.253:45965), ecb7586d3b9749e2a1f2d0ee4ebc0bfd (127.7.194.252:38797)
I20260619 02:37:06.817797 21411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9b9e58eff874edeb837df99b7714e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8cb068fd6f4175b0003d9982e641cb"
I20260619 02:37:06.817939 21411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:06.819350 21411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b9e58eff874edeb837df99b7714e25 in term 1.
I20260619 02:37:06.819825 21486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9b9e58eff874edeb837df99b7714e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd"
I20260619 02:37:06.819957 21486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:06.821249 21486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b9e58eff874edeb837df99b7714e25 in term 1.
I20260619 02:37:06.821614 21312 leader_election.cc:302] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8cb068fd6f4175b0003d9982e641cb, b9b9e58eff874edeb837df99b7714e25; no voters:
I20260619 02:37:06.822331 21519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:37:06.822624 21519 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [term 1 LEADER]: Becoming Leader. State: Replica: b9b9e58eff874edeb837df99b7714e25, State: Running, Role: LEADER
I20260619 02:37:06.822996 21519 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } }
I20260619 02:37:06.824889 21519 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9b9e58eff874edeb837df99b7714e25. Latest consensus state: current_term: 1 leader_uuid: "b9b9e58eff874edeb837df99b7714e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.825009 21519 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:06.825393 21569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:37:06.826365 21569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:37:06.833666 21486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [term 1 FOLLOWER]: Refusing update from remote peer b9b9e58eff874edeb837df99b7714e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:06.834401 21519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:06.838691 21541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9b9e58eff874edeb837df99b7714e25. Latest consensus state: current_term: 1 leader_uuid: "b9b9e58eff874edeb837df99b7714e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.838832 21541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:37:06.842391 21411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [term 1 FOLLOWER]: Refusing update from remote peer b9b9e58eff874edeb837df99b7714e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:06.842976 21519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:06.847174 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9b9e58eff874edeb837df99b7714e25. Latest consensus state: current_term: 1 leader_uuid: "b9b9e58eff874edeb837df99b7714e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.847316 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:37:06.856392 21567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9b9e58eff874edeb837df99b7714e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.856586 21567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:06.857714 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9b9e58eff874edeb837df99b7714e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.857841 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:37:06.858237 21570 mvcc.cc:205] Tried to move back new op lower bound from 7298402823502356480 to 7298402823470452736. Current Snapshot: MvccSnapshot[applied={T|T < 7298402823502356480}]
I20260619 02:37:06.859498 21519 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9b9e58eff874edeb837df99b7714e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.859622 21519 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:06.860522 21541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9b9e58eff874edeb837df99b7714e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b9e58eff874edeb837df99b7714e25" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 37137 } } peers { permanent_uuid: "0b8cb068fd6f4175b0003d9982e641cb" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 45965 } } peers { permanent_uuid: "ecb7586d3b9749e2a1f2d0ee4ebc0bfd" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 38797 } } }
I20260619 02:37:06.860644 21541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:37:06.861657 21573 mvcc.cc:205] Tried to move back new op lower bound from 7298402823502356480 to 7298402823470452736. Current Snapshot: MvccSnapshot[applied={T|T < 7298402823502356480}]
I20260619 02:37:06.862090 21571 mvcc.cc:205] Tried to move back new op lower bound from 7298402823502356480 to 7298402823470452736. Current Snapshot: MvccSnapshot[applied={T|T < 7298402823502356480}]
I20260619 02:37:06.863984 21569 catalog_manager.cc:1347] Generated new cluster ID: 31d024bdcfd7457381d068058eb327f8
I20260619 02:37:06.864061 21569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:37:06.916247 21569 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:37:06.917966 21569 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:37:06.929944 21569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9b9e58eff874edeb837df99b7714e25: Generated new TSK 0
I20260619 02:37:06.931034 21569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:37:07.725396 21552 catalog_manager.cc:1259] Loaded cluster ID: 31d024bdcfd7457381d068058eb327f8
I20260619 02:37:07.725477 21552 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd: loading cluster ID for follower catalog manager: success
I20260619 02:37:07.727241 21552 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd: acquiring CA information for follower catalog manager: success
I20260619 02:37:07.728446 21552 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ecb7586d3b9749e2a1f2d0ee4ebc0bfd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:37:07.730594 21565 catalog_manager.cc:1259] Loaded cluster ID: 31d024bdcfd7457381d068058eb327f8
I20260619 02:37:07.730654 21565 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb: loading cluster ID for follower catalog manager: success
I20260619 02:37:07.732227 21565 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb: acquiring CA information for follower catalog manager: success
I20260619 02:37:07.733292 21565 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b8cb068fd6f4175b0003d9982e641cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:37:07.994545 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 21294
I20260619 02:37:08.001298 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 21369
I20260619 02:37:08.007355 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 21444
2026-06-19T02:37:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3177 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:37:08 dist-test-slave-9tkz krb5kdc[21583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:37:08 dist-test-slave-9tkz krb5kdc[21583](info): set up 2 sockets
Jun 19 02:37:08 dist-test-slave-9tkz krb5kdc[21583](info): commencing operation
krb5kdc: starting...
W20260619 02:37:09.062012 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:37:09 dist-test-slave-9tkz krb5kdc[21583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:37:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:37:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:37:09.188468 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:42719
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:42719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:09.302209 21596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:09.302505 21596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:09.302547 21596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:09.306663 21596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:09.306735 21596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:09.306756 21596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:09.306773 21596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:09.306789 21596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:09.312165 21596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:42719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:09.313259 21596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:09.314383 21596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:37:09.323177 21596 server_base.cc:1034] running on GCE node
W20260619 02:37:09.323108 21604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:09.323113 21601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:09.323333 21602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:09.323904 21596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:09.325034 21596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:09.326195 21596 hybrid_clock.cc:648] HybridClock initialized: now 1781836629326165 us; error 45 us; skew 500 ppm
Jun 19 02:37:09 dist-test-slave-9tkz krb5kdc[21583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836629, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:09.328893 21596 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:37:09.329836 21596 webserver.cc:460] Webserver started at http://127.7.194.254:42715/ using document root <none> and password file <none>
I20260619 02:37:09.330420 21596 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:09.330478 21596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:09.330641 21596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:09.332690 21596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "074ec4bd794442f182ea9c9350793d3f"
format_stamp: "Formatted at 2026-06-19 02:37:09 on dist-test-slave-9tkz"
server_key: "3fcd958f507849fbb975240c2a2d313d"
server_key_iv: "ba9efc5fb4f9f246580b354479a5c76b"
server_key_version: "encryptionkey@0"
I20260619 02:37:09.333196 21596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "074ec4bd794442f182ea9c9350793d3f"
format_stamp: "Formatted at 2026-06-19 02:37:09 on dist-test-slave-9tkz"
server_key: "3fcd958f507849fbb975240c2a2d313d"
server_key_iv: "ba9efc5fb4f9f246580b354479a5c76b"
server_key_version: "encryptionkey@0"
I20260619 02:37:09.336889 21596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:37:09.339419 21611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:09.340742 21596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:37:09.340879 21596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "074ec4bd794442f182ea9c9350793d3f"
format_stamp: "Formatted at 2026-06-19 02:37:09 on dist-test-slave-9tkz"
server_key: "3fcd958f507849fbb975240c2a2d313d"
server_key_iv: "ba9efc5fb4f9f246580b354479a5c76b"
server_key_version: "encryptionkey@0"
I20260619 02:37:09.341012 21596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:09.352892 21596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:09.356458 21596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:09.356673 21596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:09.363938 21596 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:42719
I20260619 02:37:09.364008 21663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:42719 every 8 connection(s)
I20260619 02:37:09.364946 21596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:37:09.365240 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 21596
I20260619 02:37:09.365384 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:37:09.365782 7947 external_mini_cluster.cc:1428] Setting key 15e7bfa57a5263d1935f0e2600071b17
I20260619 02:37:09.368359 21664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:09.374164 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f: Bootstrap starting.
Jun 19 02:37:09 dist-test-slave-9tkz krb5kdc[21583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:09.376771 21664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:09.377534 21664 log.cc:824] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:09.379602 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f: No bootstrap required, opened a new log
I20260619 02:37:09.380409 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:09.367147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.7.194.254:42719)
0619 02:37:09.367607 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:09.367622 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:09.367659 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:37:09.368306 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:09.368323 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:09.368734 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:09.369092 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:09.369955 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:09.370949 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:09.371892 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:09.372373 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:09.372559 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:09.375235 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:09.375241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:09.375255 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:09.375283 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:09.377338 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:09.377893 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:09.377910 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:09.377915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:09.378049 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:09.378349 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:09.378357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:09.378358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:09.378770 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:09.378980 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:09.379291 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 02:37:09.379602 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":147,"threads_started":1}
I20260619 02:37:09.383466 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:37:09.383493 21664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ec4bd794442f182ea9c9350793d3f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42719 } }
I20260619 02:37:09.383777 21664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:09.383864 21664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074ec4bd794442f182ea9c9350793d3f, State: Initialized, Role: FOLLOWER
I20260619 02:37:09.384289 21664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ec4bd794442f182ea9c9350793d3f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42719 } }
I20260619 02:37:09.384452 21664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:37:09.384544 21664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:37:09.384649 21664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:09.385726 21664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ec4bd794442f182ea9c9350793d3f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42719 } }
I20260619 02:37:09.386080 21664 leader_election.cc:302] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 074ec4bd794442f182ea9c9350793d3f; no voters:
I20260619 02:37:09.386430 21664 leader_election.cc:288] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:37:09.386526 21669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:37:09.386849 21669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [term 1 LEADER]: Becoming Leader. State: Replica: 074ec4bd794442f182ea9c9350793d3f, State: Running, Role: LEADER
I20260619 02:37:09.387213 21669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ec4bd794442f182ea9c9350793d3f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42719 } }
I20260619 02:37:09.387661 21664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:37:09.388787 21670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "074ec4bd794442f182ea9c9350793d3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ec4bd794442f182ea9c9350793d3f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42719 } } }
I20260619 02:37:09.388911 21670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:09.389240 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 074ec4bd794442f182ea9c9350793d3f. Latest consensus state: current_term: 1 leader_uuid: "074ec4bd794442f182ea9c9350793d3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ec4bd794442f182ea9c9350793d3f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 42719 } } }
I20260619 02:37:09.389328 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:09.389484 21678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:37:09.392278 21678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:37:09.397629 21678 catalog_manager.cc:1347] Generated new cluster ID: 8da805e8a8ee4f4aa23e45c3bbb52de0
I20260619 02:37:09.397703 21678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:37:09.413659 21678 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:37:09.415024 21678 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:37:09.421411 21678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 074ec4bd794442f182ea9c9350793d3f: Generated new TSK 0
I20260619 02:37:09.422375 21678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:37:09.439883 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 21596
2026-06-19T02:37:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1431 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:37:09 dist-test-slave-9tkz krb5kdc[21691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:37:09 dist-test-slave-9tkz krb5kdc[21691](info): set up 2 sockets
Jun 19 02:37:09 dist-test-slave-9tkz krb5kdc[21691](info): commencing operation
krb5kdc: starting...
W20260619 02:37:10.482100 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:37:10 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:37:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:37:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:37:10.608947 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38323
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:10.722122 21704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:10.722414 21704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:10.722457 21704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:10.726707 21704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:10.726771 21704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:10.726791 21704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:10.726809 21704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:10.726824 21704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:10.732254 21704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:10.733404 21704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:10.734596 21704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:10.743158 21710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:10.743191 21712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:10.743167 21709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:10.743494 21704 server_base.cc:1034] running on GCE node
I20260619 02:37:10.744136 21704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:10.745333 21704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:10.746516 21704 hybrid_clock.cc:648] HybridClock initialized: now 1781836630746502 us; error 64 us; skew 500 ppm
Jun 19 02:37:10 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836630, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:10.749815 21704 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:37:10.750689 21704 webserver.cc:460] Webserver started at http://127.7.194.254:36055/ using document root <none> and password file <none>
I20260619 02:37:10.751214 21704 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:10.751272 21704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:10.751443 21704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:10.753486 21704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "0c43ede0cbc74c7282b537f93aca3a16"
format_stamp: "Formatted at 2026-06-19 02:37:10 on dist-test-slave-9tkz"
server_key: "9d388a9fbfacacb60ff139db697b3319"
server_key_iv: "00eca4dd220c99dad37fefe77213c2b0"
server_key_version: "encryptionkey@0"
I20260619 02:37:10.754089 21704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "0c43ede0cbc74c7282b537f93aca3a16"
format_stamp: "Formatted at 2026-06-19 02:37:10 on dist-test-slave-9tkz"
server_key: "9d388a9fbfacacb60ff139db697b3319"
server_key_iv: "00eca4dd220c99dad37fefe77213c2b0"
server_key_version: "encryptionkey@0"
I20260619 02:37:10.757972 21704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:37:10.760610 21719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:10.762027 21704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:37:10.762158 21704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "0c43ede0cbc74c7282b537f93aca3a16"
format_stamp: "Formatted at 2026-06-19 02:37:10 on dist-test-slave-9tkz"
server_key: "9d388a9fbfacacb60ff139db697b3319"
server_key_iv: "00eca4dd220c99dad37fefe77213c2b0"
server_key_version: "encryptionkey@0"
I20260619 02:37:10.762264 21704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:10.774025 21704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:10.777333 21704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:10.777570 21704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:10.784610 21704 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:38323
I20260619 02:37:10.784889 21771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:38323 every 8 connection(s)
I20260619 02:37:10.785662 21704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:37:10.788568 21772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:10.794219 21772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Bootstrap starting.
I20260619 02:37:10.795666 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 21704
I20260619 02:37:10.795810 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:37:10.796145 7947 external_mini_cluster.cc:1428] Setting key b712a0b59586869c25db13f143511933
I20260619 02:37:10.796974 21772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:10.797869 21772 log.cc:824] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:10.799801 21772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: No bootstrap required, opened a new log
I20260619 02:37:10.803269 21772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:10.803488 21772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:10.803548 21772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c43ede0cbc74c7282b537f93aca3a16, State: Initialized, Role: FOLLOWER
I20260619 02:37:10.804039 21772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:10.804162 21772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:37:10.804246 21772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:37:10.804356 21772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 02:37:10 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:10.805325 21772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:10.805716 21772 leader_election.cc:302] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c43ede0cbc74c7282b537f93aca3a16; no voters:
I20260619 02:37:10.806092 21772 leader_election.cc:288] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:37:10.806226 21777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:37:10.806416 21777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 1 LEADER]: Becoming Leader. State: Replica: 0c43ede0cbc74c7282b537f93aca3a16, State: Running, Role: LEADER
I20260619 02:37:10.806972 21777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:10.807440 21772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:37:10.808246 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c43ede0cbc74c7282b537f93aca3a16. Latest consensus state: current_term: 1 leader_uuid: "0c43ede0cbc74c7282b537f93aca3a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } } }
I20260619 02:37:10.808373 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:10.808691 21779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c43ede0cbc74c7282b537f93aca3a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } } }
I20260619 02:37:10.808777 21779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:10.811406 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:10.797297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.7.194.254:38323)
0619 02:37:10.797786 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:10.797801 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:10.797837 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:37:10.798558 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:10.798577 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:10.798934 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:10.799287 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:10.800142 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:10.801128 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:10.802100 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:10.802590 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:10.802755 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:10.805404 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:10.805409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:10.805422 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:10.805472 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:10.807498 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:10.808025 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:10.808031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:10.808044 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:10.808191 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:10.808551 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:10.808556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:10.808558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:10.809123 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:10.809915 (+ 792us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:10.810071 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 02:37:10.810398 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":179,"threads_started":1}
W20260619 02:37:10.813519 21792 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:37:10.813627 21792 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:37:10.813680 21786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:37:10.814546 21786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:37:10.820298 21786 catalog_manager.cc:1347] Generated new cluster ID: 460c1183fd8a4a5f919287840be21755
I20260619 02:37:10.820365 21786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:37:10.835592 21786 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:37:10.836591 21786 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:37:10.846946 21786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Generated new TSK 0
I20260619 02:37:10.847826 21786 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:37:10.884274 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38323
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:11.000977 21798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:11.001322 21798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:11.001408 21798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:11.005975 21798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:11.006096 21798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:11.006198 21798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:37:11.012076 21798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:11.013265 21798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:11.014381 21798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:11.024297 21804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:11.024350 21806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:11.024468 21798 server_base.cc:1034] running on GCE node
W20260619 02:37:11.024350 21803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:11.024896 21798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:11.025635 21798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:11.026850 21798 hybrid_clock.cc:648] HybridClock initialized: now 1781836631026812 us; error 48 us; skew 500 ppm
Jun 19 02:37:11 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836631, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:11.030094 21798 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:37:11.031052 21798 webserver.cc:460] Webserver started at http://127.7.194.193:41307/ using document root <none> and password file <none>
I20260619 02:37:11.031627 21798 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:11.031720 21798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:11.031955 21798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:11.034111 21798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "f10de6179d594e97a0b4261ef5a48c10"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "dee9f4e20a120e58dcaa9736c7e0d9ca"
server_key_iv: "c7bfc179e92ac6cf6a4a635cf1c2eeaa"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.034705 21798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "f10de6179d594e97a0b4261ef5a48c10"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "dee9f4e20a120e58dcaa9736c7e0d9ca"
server_key_iv: "c7bfc179e92ac6cf6a4a635cf1c2eeaa"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.038522 21798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:37:11.041023 21813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.042320 21798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:37:11.042462 21798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "f10de6179d594e97a0b4261ef5a48c10"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "dee9f4e20a120e58dcaa9736c7e0d9ca"
server_key_iv: "c7bfc179e92ac6cf6a4a635cf1c2eeaa"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.042577 21798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:11.074654 21798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:11.078372 21798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:11.078609 21798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:11.079257 21798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:11.080125 21798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:11.080199 21798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.080284 21798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:11.080332 21798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.089498 21798 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:37505
I20260619 02:37:11.089529 21926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:37505 every 8 connection(s)
I20260619 02:37:11.090567 21798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:37:11.091594 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 21798
I20260619 02:37:11.091727 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:37:11.092020 7947 external_mini_cluster.cc:1428] Setting key f4c3dec820382472f680bd1cedcaf3e0
Jun 19 02:37:11 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836631, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:37:11.105980 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.092873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:33479 (local address 127.7.194.254:38323)
0619 02:37:11.093009 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.093013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.094114 (+ 1101us) server_negotiation.cc:407] Connection header received
0619 02:37:11.095018 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.095021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.095074 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.095151 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:11.096421 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.096945 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.098883 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.099117 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.099202 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.102192 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:11.102195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:11.102197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:11.102221 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:11.103670 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:11.104225 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:11.104229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:11.104231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:11.104312 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:11.104668 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:11.104671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:11.104673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:11.104892 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:11.104995 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.105538 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.105721 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:37:11.106729 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.093213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:33479)
0619 02:37:11.093939 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.093982 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:11.094813 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:11.095325 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:11.095332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:11.095702 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:11.096263 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.096284 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.097085 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.097092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.098749 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.098764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.099286 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.099292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.099549 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.100247 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:11.100272 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:11.101997 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:11.103792 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:11.103798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:11.103814 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:11.104112 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:11.104402 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:11.104407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:11.104409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:11.104555 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:11.105018 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:11.105025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:11.105281 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 02:37:11.105441 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 02:37:11.105778 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":128,"threads_started":1}
I20260619 02:37:11.108215 21927 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38323
I20260619 02:37:11.108546 21927 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:11.109139 21927 heartbeater.cc:511] Master 127.7.194.254:38323 requested a full tablet report, sending...
I20260619 02:37:11.111039 21736 ts_manager.cc:194] Registered new tserver with Master: f10de6179d594e97a0b4261ef5a48c10 (127.7.194.193:37505)
I20260619 02:37:11.112805 21736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:33479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:37:11.123761 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38323
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:11.239799 21932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:11.240097 21932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:11.240144 21932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:11.244225 21932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:11.244303 21932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:11.244367 21932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:37:11.249819 21932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:11.250916 21932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:11.251917 21932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:11.262002 21938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:11.261957 21937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:11.261974 21940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:11.262348 21932 server_base.cc:1034] running on GCE node
I20260619 02:37:11.262744 21932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:11.263386 21932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:11.264580 21932 hybrid_clock.cc:648] HybridClock initialized: now 1781836631264563 us; error 27 us; skew 500 ppm
Jun 19 02:37:11 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836631, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:11.267712 21932 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:37:11.268674 21932 webserver.cc:460] Webserver started at http://127.7.194.194:45993/ using document root <none> and password file <none>
I20260619 02:37:11.269265 21932 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:11.269332 21932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:11.269876 21932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:11.272078 21932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "718f816bee234196a60df46408475c47"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "10e1266cf5c0199b340cdf83dd2dc64a"
server_key_iv: "e3bcf540b46e583f8a458b4baf552dd6"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.272687 21932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "718f816bee234196a60df46408475c47"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "10e1266cf5c0199b340cdf83dd2dc64a"
server_key_iv: "e3bcf540b46e583f8a458b4baf552dd6"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.276774 21932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:37:11.279382 21947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.280753 21932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:37:11.280854 21932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "718f816bee234196a60df46408475c47"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "10e1266cf5c0199b340cdf83dd2dc64a"
server_key_iv: "e3bcf540b46e583f8a458b4baf552dd6"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.280941 21932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:11.293286 21932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:11.296789 21932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:11.297003 21932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:11.297685 21932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:11.298545 21932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:11.298591 21932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.298631 21932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:11.298647 21932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.307633 21932 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:36529
I20260619 02:37:11.307703 22060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:36529 every 8 connection(s)
I20260619 02:37:11.308720 21932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:37:11.310570 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 21932
I20260619 02:37:11.310663 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:37:11.310992 7947 external_mini_cluster.cc:1428] Setting key 3acb0c46dfea33b11e26f5a9f707ec60
Jun 19 02:37:11 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836631, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:37:11.324869 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.310933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:50467 (local address 127.7.194.254:38323)
0619 02:37:11.311083 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.311088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.312166 (+ 1078us) server_negotiation.cc:407] Connection header received
0619 02:37:11.313664 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.313667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.313721 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.313800 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:11.315045 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.315951 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.317448 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.317949 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.318085 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.320889 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:11.320893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:11.320897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:11.320927 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:11.322615 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:11.323136 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:11.323140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:11.323141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:11.323221 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:11.323656 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:11.323660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:11.323662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:11.323865 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:11.323952 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.324623 (+ 671us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.324730 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:37:11.326043 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.311467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:50467)
0619 02:37:11.312015 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.312060 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:11.313403 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:11.313992 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:11.313999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:11.314361 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:11.314897 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.314912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.316115 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.316121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.317316 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.317325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.318098 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.318101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.318264 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.318894 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:11.318911 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:11.320675 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:11.322747 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:11.322752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:11.322764 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:11.323021 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:11.323335 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:11.323337 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:11.323339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:11.323541 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:11.323981 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:11.323989 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:11.324351 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 02:37:11.324581 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 02:37:11.324965 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":157,"threads_started":1}
I20260619 02:37:11.327301 22061 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38323
I20260619 02:37:11.327579 22061 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:11.328214 22061 heartbeater.cc:511] Master 127.7.194.254:38323 requested a full tablet report, sending...
I20260619 02:37:11.329450 21736 ts_manager.cc:194] Registered new tserver with Master: 718f816bee234196a60df46408475c47 (127.7.194.194:36529)
I20260619 02:37:11.330107 21736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:50467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:37:11.340880 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38323
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:11.455034 22066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:11.455371 22066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:11.455451 22066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:11.459777 22066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:11.459874 22066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:11.459972 22066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:37:11.465392 22066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:11.466617 22066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:11.467725 22066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:11.477638 22071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:11.477638 22074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:11.477772 22066 server_base.cc:1034] running on GCE node
W20260619 02:37:11.477663 22072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:11.478339 22066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:11.479019 22066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:11.480212 22066 hybrid_clock.cc:648] HybridClock initialized: now 1781836631480198 us; error 35 us; skew 500 ppm
Jun 19 02:37:11 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836631, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:11.485924 22066 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:37:11.486864 22066 webserver.cc:460] Webserver started at http://127.7.194.195:37329/ using document root <none> and password file <none>
I20260619 02:37:11.487507 22066 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:11.487601 22066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:11.487854 22066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:11.490070 22066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "b84820a38e1cd5805af5e2cc13de3f49"
server_key_iv: "df137df1c15f360739223fd1e3e0931e"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.490653 22066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "b84820a38e1cd5805af5e2cc13de3f49"
server_key_iv: "df137df1c15f360739223fd1e3e0931e"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.494516 22066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:37:11.497110 22081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.498481 22066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:37:11.498639 22066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "b84820a38e1cd5805af5e2cc13de3f49"
server_key_iv: "df137df1c15f360739223fd1e3e0931e"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.498777 22066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:11.512522 22066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:11.515750 22066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:11.515964 22066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:11.516587 22066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:11.517450 22066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:11.517535 22066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.517638 22066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:11.517690 22066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.526999 22066 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:33427
I20260619 02:37:11.527036 22194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:33427 every 8 connection(s)
I20260619 02:37:11.527915 22066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:37:11 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836631, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:11.538149 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 22066
I20260619 02:37:11.538277 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:37:11.538581 7947 external_mini_cluster.cc:1428] Setting key 92620a89a436ffaa70dfc8e639f41563
I20260619 02:37:11.543120 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.529877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:51127 (local address 127.7.194.254:38323)
0619 02:37:11.530027 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.530030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.530888 (+ 858us) server_negotiation.cc:407] Connection header received
0619 02:37:11.532239 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.532243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.532303 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.532410 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:11.533734 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.534316 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.535488 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.535740 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.535862 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.538866 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:11.538870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:11.538873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:11.538905 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:11.540757 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:11.541303 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:11.541305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:11.541307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:11.541383 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:11.541772 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:11.541776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:11.541777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:11.541981 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:11.542124 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.542880 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.542990 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:37:11.544100 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.530260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:51127)
0619 02:37:11.530724 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.530767 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:11.532012 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:11.532601 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:11.532609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:11.532978 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:11.533489 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.533502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.534451 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.534460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.535356 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.535365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.535904 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.535908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.536086 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.536694 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:11.536710 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:11.538641 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:11.540865 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:11.540870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:11.540883 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:11.541214 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:11.541443 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:11.541445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:11.541447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:11.541646 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:11.542339 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:11.542345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:11.542606 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 02:37:11.542778 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 02:37:11.543084 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260619 02:37:11.545295 22195 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38323
I20260619 02:37:11.545614 22195 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:11.546229 22195 heartbeater.cc:511] Master 127.7.194.254:38323 requested a full tablet report, sending...
I20260619 02:37:11.547547 21736 ts_manager.cc:194] Registered new tserver with Master: 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427)
I20260619 02:37:11.548146 21736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:51127
I20260619 02:37:11.552664 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:37:11.561985 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.554686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.7.194.254:38323)
0619 02:37:11.554835 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.554840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.554995 (+ 155us) server_negotiation.cc:407] Connection header received
0619 02:37:11.555100 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.555102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.555142 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.555206 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:11.555918 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.556446 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.557229 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.557469 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.557619 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.558873 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:11.558876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:11.558878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:11.558902 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:11.560365 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:11.560723 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:11.560726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:11.560727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:11.560801 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:11.561018 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:11.561020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:11.561021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:11.561256 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:11.561363 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.561690 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.561799 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:37:11.565084 21736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:37:11.568312 21736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:37:11.583920 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.577242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:37505 (local address 127.0.0.1:39428)
0619 02:37:11.578261 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.578317 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:11.578465 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:11.579178 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:11.579181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:11.579199 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:11.579342 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.579350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.580847 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.580852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.582226 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.582237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.583347 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.583353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.583475 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.583513 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:37:11.583630 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:37:11.583729 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":31,"spinlock_wait_cycles":2688,"thread_start_us":61,"threads_started":1}
I20260619 02:37:11.583981 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.577239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:36529 (local address 127.0.0.1:41628)
0619 02:37:11.578338 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.578353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:11.578470 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:11.579061 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:11.579067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:11.579090 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:11.579254 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.579261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.580737 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.580740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.582603 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.582610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.583582 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.583585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.583669 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.583693 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:37:11.583769 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:37:11.583835 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":163,"spinlock_wait_cycles":5120,"thread_start_us":132,"threads_started":1}
I20260619 02:37:11.584645 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.577813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.7.194.193:37505)
0619 02:37:11.578749 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.578763 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.578784 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:37:11.578847 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.578851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.579024 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.579484 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:11.579634 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.580706 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.582357 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.583214 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.583328 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.583506 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.584149 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.584507 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":171,"threads_started":1}
I20260619 02:37:11.584640 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.577384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.7.194.194:36529)
0619 02:37:11.578454 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.578469 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.578496 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:37:11.578665 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.578674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.578898 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.579063 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:11.579367 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.580586 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.582716 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.583298 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.583406 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.583562 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.583823 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.583923 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":161,"threads_started":1}
I20260619 02:37:11.585213 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.577801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:33427 (local address 127.0.0.1:55330)
0619 02:37:11.578651 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.578665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:11.578755 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:11.579262 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:11.579264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:11.579277 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:11.580790 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.580804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.582181 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.582187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.583339 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.583349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.584801 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.584804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.584912 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.584935 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:37:11.585022 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:37:11.585095 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":1397,"thread_start_us":118,"threads_started":1}
I20260619 02:37:11.585651 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.577960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.7.194.195:33427)
0619 02:37:11.578776 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.578791 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.578815 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:37:11.578895 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.578898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.579106 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.579257 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:11.580929 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.582050 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.583475 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.584497 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.584601 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.584740 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.585378 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.585473 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":131,"threads_started":1}
I20260619 02:37:11.587396 21995 tablet_service.cc:1467] Processing CreateTablet for tablet 693bdf6d1a544ec2a2eb5b577221058a (DEFAULT_TABLE table=test-table [id=a6223dc8bdd34deb990327c4e5b8f64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:11.587401 21861 tablet_service.cc:1467] Processing CreateTablet for tablet 693bdf6d1a544ec2a2eb5b577221058a (DEFAULT_TABLE table=test-table [id=a6223dc8bdd34deb990327c4e5b8f64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:11.588474 21861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693bdf6d1a544ec2a2eb5b577221058a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:11.588485 21995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693bdf6d1a544ec2a2eb5b577221058a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:11.590523 22129 tablet_service.cc:1467] Processing CreateTablet for tablet 693bdf6d1a544ec2a2eb5b577221058a (DEFAULT_TABLE table=test-table [id=a6223dc8bdd34deb990327c4e5b8f64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:11.591589 22129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693bdf6d1a544ec2a2eb5b577221058a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:11.595062 22212 tablet_bootstrap.cc:492] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Bootstrap starting.
I20260619 02:37:11.596370 22213 tablet_bootstrap.cc:492] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Bootstrap starting.
I20260619 02:37:11.597805 22212 tablet_bootstrap.cc:654] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:11.598636 22213 tablet_bootstrap.cc:654] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:11.598675 22212 log.cc:824] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:11.599581 22214 tablet_bootstrap.cc:492] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: Bootstrap starting.
I20260619 02:37:11.599874 22213 log.cc:824] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:11.601171 22212 tablet_bootstrap.cc:492] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: No bootstrap required, opened a new log
I20260619 02:37:11.601377 22212 ts_tablet_manager.cc:1397] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 02:37:11.601732 22213 tablet_bootstrap.cc:492] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: No bootstrap required, opened a new log
I20260619 02:37:11.601936 22213 ts_tablet_manager.cc:1397] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:37:11.602319 22214 tablet_bootstrap.cc:654] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:11.603176 22214 log.cc:824] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:11.605269 22214 tablet_bootstrap.cc:492] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: No bootstrap required, opened a new log
I20260619 02:37:11.605228 22213 raft_consensus.cc:348] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:11.605475 22214 ts_tablet_manager.cc:1397] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:37:11.605505 22213 raft_consensus.cc:374] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:11.605592 22213 raft_consensus.cc:729] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10de6179d594e97a0b4261ef5a48c10, State: Initialized, Role: FOLLOWER
I20260619 02:37:11.605479 22212 raft_consensus.cc:348] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:11.605772 22212 raft_consensus.cc:374] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:11.605831 22212 raft_consensus.cc:729] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718f816bee234196a60df46408475c47, State: Initialized, Role: FOLLOWER
I20260619 02:37:11.606127 22213 consensus_queue.cc:260] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:11.606335 22212 consensus_queue.cc:260] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:11.607156 22061 heartbeater.cc:503] Master 127.7.194.254:38323 was elected leader, sending a full tablet report...
I20260619 02:37:11.607380 22213 ts_tablet_manager.cc:1428] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260619 02:37:11.607446 22212 ts_tablet_manager.cc:1428] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:37:11.607687 21927 heartbeater.cc:503] Master 127.7.194.254:38323 was elected leader, sending a full tablet report...
I20260619 02:37:11.610594 22214 raft_consensus.cc:348] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:11.610924 22214 raft_consensus.cc:374] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:11.610998 22214 raft_consensus.cc:729] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce53c8b4ce8468e94b0f7a568d5b06d, State: Initialized, Role: FOLLOWER
I20260619 02:37:11.611490 22214 consensus_queue.cc:260] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:11.612125 22195 heartbeater.cc:503] Master 127.7.194.254:38323 was elected leader, sending a full tablet report...
I20260619 02:37:11.612529 22214 ts_tablet_manager.cc:1428] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 02:37:11.653625 22218 raft_consensus.cc:482] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:37:11.653920 22218 raft_consensus.cc:504] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:11.655225 22218 leader_election.cc:288] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 718f816bee234196a60df46408475c47 (127.7.194.194:36529), 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427)
I20260619 02:37:11.659298 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.655480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:33427 (local address 127.7.194.193:33117)
0619 02:37:11.655950 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.655965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:11.656099 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:11.656356 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:11.656358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:11.656378 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:11.656515 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.656521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.657293 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.657296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.658174 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.658182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.658822 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.658826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.658933 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.658960 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 02:37:11.659042 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:37:11.659114 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":47,"thread_start_us":85,"threads_started":1}
I20260619 02:37:11.659300 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.655706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:33117 (local address 127.7.194.195:33427)
0619 02:37:11.655835 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.655841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.656076 (+ 235us) server_negotiation.cc:407] Connection header received
0619 02:37:11.656202 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.656206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.656256 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.656332 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:11.656635 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.657180 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.658298 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.658706 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.658806 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.658848 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.659075 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.659174 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:37:11.659360 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.655482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:36529 (local address 127.7.194.193:34519)
0619 02:37:11.655833 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.655850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:11.655962 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:11.656234 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:11.656237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:11.656253 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:11.656416 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.656424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.657195 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.657198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.658181 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:11.658190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.658948 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.658951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:11.659053 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.659083 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:37:11.659159 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:37:11.659223 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":66,"thread_start_us":110,"threads_started":1}
I20260619 02:37:11.659386 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.655609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:34519 (local address 127.7.194.194:36529)
0619 02:37:11.655751 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.655758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.655937 (+ 179us) server_negotiation.cc:407] Connection header received
0619 02:37:11.656073 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.656077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.656133 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.656220 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:11.656549 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.657085 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.658345 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.658823 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.658936 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.658980 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.659190 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.659257 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:37:11.659999 22149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693bdf6d1a544ec2a2eb5b577221058a" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" is_pre_election: true
I20260619 02:37:11.660070 22015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693bdf6d1a544ec2a2eb5b577221058a" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718f816bee234196a60df46408475c47" is_pre_election: true
I20260619 02:37:11.660369 22149 raft_consensus.cc:2454] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 0.
I20260619 02:37:11.660369 22015 raft_consensus.cc:2454] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 0.
I20260619 02:37:11.660985 21817 leader_election.cc:302] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 718f816bee234196a60df46408475c47, 7ce53c8b4ce8468e94b0f7a568d5b06d, f10de6179d594e97a0b4261ef5a48c10; no voters:
I20260619 02:37:11.661337 22218 raft_consensus.cc:2781] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:37:11.661437 22218 raft_consensus.cc:482] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:37:11.661500 22218 raft_consensus.cc:3037] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:11.662669 22218 raft_consensus.cc:504] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:11.663046 22218 leader_election.cc:288] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 1 election: Requested vote from peers 718f816bee234196a60df46408475c47 (127.7.194.194:36529), 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427)
I20260619 02:37:11.663488 22149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693bdf6d1a544ec2a2eb5b577221058a" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d"
I20260619 02:37:11.663488 22015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693bdf6d1a544ec2a2eb5b577221058a" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718f816bee234196a60df46408475c47"
I20260619 02:37:11.663630 22149 raft_consensus.cc:3037] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:11.663630 22015 raft_consensus.cc:3037] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:11.664968 22015 raft_consensus.cc:2454] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 1.
I20260619 02:37:11.664968 22149 raft_consensus.cc:2454] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 1.
I20260619 02:37:11.665256 21817 leader_election.cc:302] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718f816bee234196a60df46408475c47, f10de6179d594e97a0b4261ef5a48c10; no voters:
I20260619 02:37:11.665495 22218 raft_consensus.cc:2781] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:37:11.665788 22218 raft_consensus.cc:686] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 1 LEADER]: Becoming Leader. State: Replica: f10de6179d594e97a0b4261ef5a48c10, State: Running, Role: LEADER
I20260619 02:37:11.666175 22218 consensus_queue.cc:237] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:11.669409 21736 catalog_manager.cc:5495] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 reported cstate change: term changed from 0 to 1, leader changed from <none> to f10de6179d594e97a0b4261ef5a48c10 (127.7.194.193). New cstate: current_term: 1 leader_uuid: "f10de6179d594e97a0b4261ef5a48c10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } health_report { overall_health: UNKNOWN } } }
I20260619 02:37:11.682179 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.678492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.7.194.193:37505)
0619 02:37:11.678643 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:11.678648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:11.678663 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:37:11.678823 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:11.678825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:11.678881 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:11.678991 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:37:11.679241 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.679756 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.680636 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:11.680859 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:11.680968 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:11.681289 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:37:11.681830 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:37:11.681980 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:11.682010 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 02:37:11.682068 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:37:11.688378 22149 raft_consensus.cc:1261] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 1 FOLLOWER]: Refusing update from remote peer f10de6179d594e97a0b4261ef5a48c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:11.688375 22015 raft_consensus.cc:1261] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Refusing update from remote peer f10de6179d594e97a0b4261ef5a48c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:11.689121 22223 consensus_queue.cc:1035] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:11.689316 22218 consensus_queue.cc:1035] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:11.697350 22228 mvcc.cc:205] Tried to move back new op lower bound from 7298402843388198912 to 7298402843307724800. Current Snapshot: MvccSnapshot[applied={T|T < 7298402843388198912}]
I20260619 02:37:11.706245 21735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:11.706444 21735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:11.709252 21735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Sending DeleteTablet for 3 replicas of tablet 693bdf6d1a544ec2a2eb5b577221058a
I20260619 02:37:11.710145 21861 tablet_service.cc:1514] Processing DeleteTablet for tablet 693bdf6d1a544ec2a2eb5b577221058a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:11 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:39428
I20260619 02:37:11.710141 21995 tablet_service.cc:1514] Processing DeleteTablet for tablet 693bdf6d1a544ec2a2eb5b577221058a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:11 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:41628
I20260619 02:37:11.710474 22129 tablet_service.cc:1514] Processing DeleteTablet for tablet 693bdf6d1a544ec2a2eb5b577221058a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:11 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:55330
I20260619 02:37:11.710913 22236 tablet_replica.cc:330] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: stopping tablet replica
I20260619 02:37:11.710952 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 21798
I20260619 02:37:11.712304 22238 tablet_replica.cc:330] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: stopping tablet replica
I20260619 02:37:11.712647 22238 raft_consensus.cc:2229] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:11.712983 22238 raft_consensus.cc:2258] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:11.713405 22237 tablet_replica.cc:330] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: stopping tablet replica
I20260619 02:37:11.713783 22237 raft_consensus.cc:2229] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:11.714087 22237 raft_consensus.cc:2258] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:11.714779 22238 ts_tablet_manager.cc:1905] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:37:11.715580 22237 ts_tablet_manager.cc:1905] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:37:11.717538 21720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:37505 (error 108)
I20260619 02:37:11.717700 22238 ts_tablet_manager.cc:1918] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:11.717784 22238 log.cc:1196] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/693bdf6d1a544ec2a2eb5b577221058a
I20260619 02:37:11.717856 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 21932
I20260619 02:37:11.718122 22238 ts_tablet_manager.cc:1939] T 693bdf6d1a544ec2a2eb5b577221058a P 7ce53c8b4ce8468e94b0f7a568d5b06d: Deleting consensus metadata
I20260619 02:37:11.719350 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.719017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:37505 (local address 127.0.0.1:39438)
0619 02:37:11.719180 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.719259 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 02:37:11.719552 21721 catalog_manager.cc:4841] TS 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427): tablet 693bdf6d1a544ec2a2eb5b577221058a (table test-table [id=a6223dc8bdd34deb990327c4e5b8f64b]) successfully deleted
W20260619 02:37:11.719537 21720 catalog_manager.cc:4584] TS f10de6179d594e97a0b4261ef5a48c10 (127.7.194.193:37505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 693bdf6d1a544ec2a2eb5b577221058a: Network error: Client connection negotiation failed: client connection to 127.7.194.193:37505: connect: Connection refused (error 111)
I20260619 02:37:11.723978 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 22066
I20260619 02:37:11.725184 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:11.724136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:36529 (local address 127.0.0.1:41642)
0619 02:37:11.725047 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:11.725106 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.194:36529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":837}
I20260619 02:37:11.730016 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 21704
I20260619 02:37:11.736550 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38323
--webserver_interface=127.7.194.254
--webserver_port=36055
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:11.849496 22240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:11.849874 22240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:11.849916 22240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:11.854228 22240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:11.854290 22240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:11.854310 22240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:11.854327 22240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:11.854342 22240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:11.859738 22240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=36055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:11.860867 22240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:11.862061 22240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:11.875414 22249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:11.875414 22247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:11.875456 22246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:11.875713 22240 server_base.cc:1034] running on GCE node
I20260619 02:37:11.876641 22240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:11.877879 22240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:11.879067 22240 hybrid_clock.cc:648] HybridClock initialized: now 1781836631879053 us; error 27 us; skew 500 ppm
Jun 19 02:37:11 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836631, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:11.881927 22240 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:37:11.882956 22240 webserver.cc:460] Webserver started at http://127.7.194.254:36055/ using document root <none> and password file <none>
I20260619 02:37:11.883558 22240 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:11.883680 22240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:11.887753 22240 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:37:11.890065 22256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:11.891330 22240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:37:11.891476 22240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "0c43ede0cbc74c7282b537f93aca3a16"
format_stamp: "Formatted at 2026-06-19 02:37:10 on dist-test-slave-9tkz"
server_key: "9d388a9fbfacacb60ff139db697b3319"
server_key_iv: "00eca4dd220c99dad37fefe77213c2b0"
server_key_version: "encryptionkey@0"
I20260619 02:37:11.892109 22240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:11.905206 22240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:11.908236 22240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:11.908456 22240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:11.916023 22240 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:38323
I20260619 02:37:11.916342 22308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:38323 every 8 connection(s)
I20260619 02:37:11.917346 22240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:37:11.921969 22309 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:37:11.923481 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 22240
I20260619 02:37:11.923959 22309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Bootstrap starting.
I20260619 02:37:11.923988 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:37505
--local_ip_for_outbound_sockets=127.7.194.193
--tserver_master_addrs=127.7.194.254:38323
--webserver_port=41307
--webserver_interface=127.7.194.193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:11.942487 22309 log.cc:824] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:11.951512 22309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:37:11.951967 22309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Bootstrap complete.
I20260619 02:37:11.956483 22309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:11.956889 22309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c43ede0cbc74c7282b537f93aca3a16, State: Initialized, Role: FOLLOWER
I20260619 02:37:11.957388 22309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:11.957501 22309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:37:11.957618 22309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:37:11.957738 22309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:37:11.959813 22309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:11.960180 22309 leader_election.cc:302] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c43ede0cbc74c7282b537f93aca3a16; no voters:
I20260619 02:37:11.960515 22309 leader_election.cc:288] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:37:11.960656 22313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:37:11.960906 22313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 2 LEADER]: Becoming Leader. State: Replica: 0c43ede0cbc74c7282b537f93aca3a16, State: Running, Role: LEADER
I20260619 02:37:11.961229 22313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:11.961843 22309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:37:11.962657 22315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c43ede0cbc74c7282b537f93aca3a16. Latest consensus state: current_term: 2 leader_uuid: "0c43ede0cbc74c7282b537f93aca3a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } } }
I20260619 02:37:11.964187 22314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0c43ede0cbc74c7282b537f93aca3a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } } }
I20260619 02:37:11.964289 22314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:11.964365 22315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:11.966732 22329 catalog_manager.cc:1259] Loaded cluster ID: 460c1183fd8a4a5f919287840be21755
I20260619 02:37:11.966809 22329 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: loading cluster ID for follower catalog manager: success
W20260619 02:37:11.967592 22329 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260619 02:37:11.967675 22329 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260619 02:37:11.967730 22330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:37:11.969792 22330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:37:11.970203 22330 catalog_manager.cc:1259] Loaded cluster ID: 460c1183fd8a4a5f919287840be21755
I20260619 02:37:11.970286 22330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:37:11.971055 22330 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 02:37:11.973869 22330 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 02:37:11.978796 22330 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:37:11.979458 22330 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Loaded TSK: 0
I20260619 02:37:11.980432 22330 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:37:12.052832 22311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:12.053175 22311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:12.053238 22311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:12.057456 22311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:12.057525 22311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:12.057687 22311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:37:12.063503 22311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:37505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=41307
--tserver_master_addrs=127.7.194.254:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:12.064553 22311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:12.065632 22311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:12.076303 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:12.076303 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:12.076297 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:12.076570 22311 server_base.cc:1034] running on GCE node
I20260619 02:37:12.077070 22311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:12.077795 22311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:12.078994 22311 hybrid_clock.cc:648] HybridClock initialized: now 1781836632078966 us; error 49 us; skew 500 ppm
Jun 19 02:37:12 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836632, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:12.082048 22311 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:37:12.083138 22311 webserver.cc:460] Webserver started at http://127.7.194.193:41307/ using document root <none> and password file <none>
I20260619 02:37:12.083700 22311 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:12.083802 22311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:12.087929 22311 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 02:37:12.090163 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:12.091504 22311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:37:12.091600 22311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "f10de6179d594e97a0b4261ef5a48c10"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "dee9f4e20a120e58dcaa9736c7e0d9ca"
server_key_iv: "c7bfc179e92ac6cf6a4a635cf1c2eeaa"
server_key_version: "encryptionkey@0"
I20260619 02:37:12.092173 22311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:12.120960 22311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:12.124259 22311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:12.124476 22311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:12.125125 22311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:12.126273 22354 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:37:12.130707 22311 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:37:12.130770 22311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 02:37:12.130842 22311 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:37:12.132570 22311 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:37:12.132623 22311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 02:37:12.132957 22354 tablet_bootstrap.cc:492] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Bootstrap starting.
I20260619 02:37:12.143491 22311 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:37505
I20260619 02:37:12.143579 22461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:37505 every 8 connection(s)
I20260619 02:37:12.144513 22311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:37:12.151525 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 22311
I20260619 02:37:12.151975 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:36529
--local_ip_for_outbound_sockets=127.7.194.194
--tserver_master_addrs=127.7.194.254:38323
--webserver_port=45993
--webserver_interface=127.7.194.194
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:12.152364 22354 log.cc:824] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:37:12 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836632, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:12.161623 22354 tablet_bootstrap.cc:492] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:37:12.162088 22354 tablet_bootstrap.cc:492] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Bootstrap complete.
I20260619 02:37:12.163640 22354 ts_tablet_manager.cc:1397] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Time spent bootstrapping tablet: real 0.031s user 0.030s sys 0.000s
I20260619 02:37:12.163800 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.146592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:45541)
0619 02:37:12.147244 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:12.147297 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:12.148125 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:12.149644 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:12.149652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:12.150029 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:12.150506 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.150521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.152348 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.152352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.153489 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.153501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.154512 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.154515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.154663 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.155277 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:12.155293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:12.156855 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:12.159558 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:12.159567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:12.159585 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:12.159992 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:12.160555 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:12.160558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:12.160560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:12.160755 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:12.161615 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:12.161621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:12.162073 (+ 452us) client_negotiation.cc:769] Sending connection context
0619 02:37:12.162284 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 02:37:12.162699 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":145,"threads_started":1}
I20260619 02:37:12.164335 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.146681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:45541 (local address 127.7.194.254:38323)
0619 02:37:12.147140 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:12.147164 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:12.147444 (+ 280us) server_negotiation.cc:407] Connection header received
0619 02:37:12.148644 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:12.148666 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:12.149140 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:12.149543 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:12.150690 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.152162 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.153655 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.154358 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.154525 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.157083 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:12.157089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:12.157104 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:12.157139 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:12.159411 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.160163 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.160172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.160181 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.160437 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.160868 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.160873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.160875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.161464 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:12.161653 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:12.162465 (+ 812us) server_negotiation.cc:299] Negotiation successful
0619 02:37:12.162874 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":127,"threads_started":1}
I20260619 02:37:12.166370 22354 raft_consensus.cc:348] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:12.166837 22462 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38323
I20260619 02:37:12.166940 22354 raft_consensus.cc:729] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10de6179d594e97a0b4261ef5a48c10, State: Initialized, Role: FOLLOWER
I20260619 02:37:12.167089 22462 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:12.167474 22354 consensus_queue.cc:260] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } } peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } }
I20260619 02:37:12.167636 22462 heartbeater.cc:511] Master 127.7.194.254:38323 requested a full tablet report, sending...
I20260619 02:37:12.168412 22354 ts_tablet_manager.cc:1428] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:37:12.170264 22273 ts_manager.cc:194] Registered new tserver with Master: f10de6179d594e97a0b4261ef5a48c10 (127.7.194.193:37505)
I20260619 02:37:12.173640 22273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:45541
I20260619 02:37:12.176278 22462 heartbeater.cc:503] Master 127.7.194.254:38323 was elected leader, sending a full tablet report...
Jun 19 02:37:12 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836631, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:37:12.183961 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.173130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.7.194.193:37505)
0619 02:37:12.173508 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:12.173519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:12.173927 (+ 408us) server_negotiation.cc:407] Connection header received
0619 02:37:12.174124 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:12.174130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:12.174330 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:12.174457 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:12.175138 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.175842 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.176855 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.177373 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.177490 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.180261 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:12.180269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:12.180278 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:12.180300 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:12.181901 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.182469 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.182476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.182479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.182588 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.182953 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.182960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.182962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.183233 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:12.183336 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:12.183694 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 02:37:12.183834 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
I20260619 02:37:12.184003 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.173030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:37505 (local address 127.0.0.1:39446)
0619 02:37:12.173787 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:12.173831 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:12.173968 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:12.174525 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:12.174531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:12.174903 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:12.175009 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.175016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.175956 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.175960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.176711 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.176723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.177500 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.177504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.177686 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.178386 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:12.178401 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:12.180092 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:12.182009 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:12.182019 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:12.182025 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:12.182338 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:12.182688 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:12.182691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:12.182693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:12.182833 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:12.183345 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:12.183351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:12.183476 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 02:37:12.183626 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 02:37:12.183827 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":142,"threads_started":1}
I20260619 02:37:12.185022 22396 tablet_service.cc:1514] Processing DeleteTablet for tablet 693bdf6d1a544ec2a2eb5b577221058a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:11 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:39446
I20260619 02:37:12.185883 22471 tablet_replica.cc:330] stopping tablet replica
I20260619 02:37:12.186213 22471 raft_consensus.cc:2229] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:12.186465 22471 raft_consensus.cc:2258] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:12.187424 22471 ts_tablet_manager.cc:1905] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:37:12.191097 22471 ts_tablet_manager.cc:1918] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:12.191176 22471 log.cc:1196] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/693bdf6d1a544ec2a2eb5b577221058a
I20260619 02:37:12.191433 22471 ts_tablet_manager.cc:1939] T 693bdf6d1a544ec2a2eb5b577221058a P f10de6179d594e97a0b4261ef5a48c10: Deleting consensus metadata
I20260619 02:37:12.193183 22257 catalog_manager.cc:4841] TS f10de6179d594e97a0b4261ef5a48c10 (127.7.194.193:37505): tablet 693bdf6d1a544ec2a2eb5b577221058a (table test-table [id=a6223dc8bdd34deb990327c4e5b8f64b]) successfully deleted
W20260619 02:37:12.273120 22467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:12.273361 22467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:12.273404 22467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:12.277390 22467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:12.277444 22467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:12.277500 22467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:37:12.282742 22467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:36529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=45993
--tserver_master_addrs=127.7.194.254:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:12.283861 22467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:12.284823 22467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:12.294258 22477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:12.294517 22467 server_base.cc:1034] running on GCE node
W20260619 02:37:12.294258 22476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:12.294296 22479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:12.295080 22467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:12.295742 22467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:12.296921 22467 hybrid_clock.cc:648] HybridClock initialized: now 1781836632296905 us; error 42 us; skew 500 ppm
Jun 19 02:37:12 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836632, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:12.299721 22467 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:37:12.300670 22467 webserver.cc:460] Webserver started at http://127.7.194.194:45993/ using document root <none> and password file <none>
I20260619 02:37:12.301266 22467 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:12.301338 22467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:12.305142 22467 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:37:12.307237 22486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:12.308468 22467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:37:12.308558 22467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "718f816bee234196a60df46408475c47"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "10e1266cf5c0199b340cdf83dd2dc64a"
server_key_iv: "e3bcf540b46e583f8a458b4baf552dd6"
server_key_version: "encryptionkey@0"
I20260619 02:37:12.309063 22467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:12.332096 22467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:12.335238 22467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:12.335395 22467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:12.335935 22467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:12.336966 22493 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 02:37:12.340888 22493 ts_tablet_manager.cc:1852] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 02:37:12.341287 22493 ts_tablet_manager.cc:1905] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:37:12.345256 22493 ts_tablet_manager.cc:1918] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:12.345355 22493 log.cc:1196] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/693bdf6d1a544ec2a2eb5b577221058a
I20260619 02:37:12.345750 22493 ts_tablet_manager.cc:1939] T 693bdf6d1a544ec2a2eb5b577221058a P 718f816bee234196a60df46408475c47: Deleting consensus metadata
I20260619 02:37:12.346091 22467 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 02:37:12.346158 22467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260619 02:37:12.346246 22467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:12.346283 22467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:12.355660 22467 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:36529
I20260619 02:37:12.355791 22600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:36529 every 8 connection(s)
I20260619 02:37:12.356637 22467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:37:12.361914 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 22467
I20260619 02:37:12.362429 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:33427
--local_ip_for_outbound_sockets=127.7.194.195
--tserver_master_addrs=127.7.194.254:38323
--webserver_port=37329
--webserver_interface=127.7.194.195
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:37:12 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836632, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:12.370965 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.358553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:44443 (local address 127.7.194.254:38323)
0619 02:37:12.358749 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:12.358753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:12.359677 (+ 924us) server_negotiation.cc:407] Connection header received
0619 02:37:12.360687 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:12.360690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:12.360743 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:12.360833 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:12.362118 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.362859 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.364282 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.364537 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.364654 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.367207 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:12.367211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:12.367213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:12.367241 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:12.368770 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.369282 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.369286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.369288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.369380 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.369799 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.369803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.369805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.369996 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:12.370139 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:12.370656 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 02:37:12.370789 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 02:37:12.371781 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.359034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:44443)
0619 02:37:12.359512 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:12.359567 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:12.360445 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:12.361005 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:12.361011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:12.361373 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:12.361916 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.361930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.363014 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.363020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.364165 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.364174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.364691 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.364695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.364876 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.365525 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:12.365584 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:12.367036 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:12.368890 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:12.368897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:12.368911 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:12.369177 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:12.369480 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:12.369483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:12.369485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:12.369701 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:12.370132 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:12.370139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:12.370399 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 02:37:12.370600 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 02:37:12.370921 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":136,"threads_started":1}
I20260619 02:37:12.373023 22601 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38323
I20260619 02:37:12.373327 22601 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:12.373893 22601 heartbeater.cc:511] Master 127.7.194.254:38323 requested a full tablet report, sending...
I20260619 02:37:12.375005 22273 ts_manager.cc:194] Registered new tserver with Master: 718f816bee234196a60df46408475c47 (127.7.194.194:36529)
I20260619 02:37:12.375814 22273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:44443
W20260619 02:37:12.481106 22604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:12.481392 22604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:12.481436 22604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:12.485769 22604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:12.485842 22604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:12.485904 22604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:37:12.491400 22604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:33427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=37329
--tserver_master_addrs=127.7.194.254:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:12.492537 22604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:12.493603 22604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:12.503536 22610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:12.503533 22612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:12.503717 22604 server_base.cc:1034] running on GCE node
W20260619 02:37:12.503533 22609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:12.504313 22604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:12.504989 22604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:12.506150 22604 hybrid_clock.cc:648] HybridClock initialized: now 1781836632506135 us; error 30 us; skew 500 ppm
Jun 19 02:37:12 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836632, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:12.509042 22604 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:37:12.510017 22604 webserver.cc:460] Webserver started at http://127.7.194.195:37329/ using document root <none> and password file <none>
I20260619 02:37:12.510587 22604 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:12.510694 22604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:12.514928 22604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:37:12.517264 22619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:12.518692 22604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:37:12.518831 22604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "b84820a38e1cd5805af5e2cc13de3f49"
server_key_iv: "df137df1c15f360739223fd1e3e0931e"
server_key_version: "encryptionkey@0"
I20260619 02:37:12.519405 22604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:12.532150 22604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:12.535305 22604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:12.535516 22604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:12.536119 22604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:12.536950 22604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:12.537029 22604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:12.537102 22604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:12.537153 22604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:12.546200 22604 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:33427
I20260619 02:37:12.546224 22732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:33427 every 8 connection(s)
I20260619 02:37:12.547212 22604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:37:12.549355 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 22604
Jun 19 02:37:12 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836632, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:12.558982 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.550054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.7.194.254:38323)
0619 02:37:12.550367 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:12.550371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:12.550505 (+ 134us) server_negotiation.cc:407] Connection header received
0619 02:37:12.550671 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:12.550675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:12.550729 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:12.550837 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:12.551650 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.552481 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.553438 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.553851 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.553962 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.555378 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:12.555382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:12.555385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:12.555412 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:12.557276 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.557777 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.557780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.557782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.557859 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.558139 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.558141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.558142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.558290 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:12.558539 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:12.558671 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:37:12.558790 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":75,"threads_started":1}
I20260619 02:37:12.562188 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.549055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:40915 (local address 127.7.194.254:38323)
0619 02:37:12.549227 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:12.549232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:12.550854 (+ 1622us) server_negotiation.cc:407] Connection header received
0619 02:37:12.551869 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:12.551871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:12.551909 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:12.551970 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:12.553360 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.553964 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.555277 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.555501 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.555611 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.558289 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:12.558292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:12.558294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:12.558315 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:12.559876 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.560446 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.560449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.560450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.560530 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:12.560914 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:12.560917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:12.560919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:12.561097 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:12.561204 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:12.561919 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 02:37:12.562032 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:37:12.563164 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.549617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:40915)
0619 02:37:12.550713 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:12.550772 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:12.551651 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:12.552155 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:12.552164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:12.552728 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:12.553217 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.553231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.554090 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.554096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.555150 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.555159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.555648 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.555652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.555831 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.556329 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:12.556344 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:12.558120 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:12.559993 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:12.560006 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:12.560024 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:12.560355 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:12.560624 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:12.560626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:12.560628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:12.560804 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:12.561225 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:12.561233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:12.561590 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 02:37:12.561865 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 02:37:12.562210 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":710,"threads_started":1}
I20260619 02:37:12.564241 22733 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38323
I20260619 02:37:12.564491 22733 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:12.564975 22733 heartbeater.cc:511] Master 127.7.194.254:38323 requested a full tablet report, sending...
I20260619 02:37:12.566301 22273 ts_manager.cc:194] Registered new tserver with Master: 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427)
I20260619 02:37:12.566879 22273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:40915
I20260619 02:37:12.572569 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:37:12.576748 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.573218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.7.194.254:38323)
0619 02:37:12.573354 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:12.573358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:12.573656 (+ 298us) server_negotiation.cc:407] Connection header received
0619 02:37:12.573778 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:12.573780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:12.573870 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:12.573957 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:37:12.574161 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.574662 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.575502 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.575701 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.575776 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.576021 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:37:12.576378 (+ 357us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:37:12.576463 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:12.576541 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 02:37:12.576586 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:37:12.577342 22273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:37:12.580687 22273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:37:12.590832 22396 tablet_service.cc:1467] Processing CreateTablet for tablet 72f3ef36eb8f4509b8afd322449835f0 (DEFAULT_TABLE table=test-table [id=55a47fe920d643348ef329c3216cb786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:12.591544 22396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f3ef36eb8f4509b8afd322449835f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:12.594472 22354 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Bootstrap starting.
I20260619 02:37:12.595207 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.589259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:33427 (local address 127.0.0.1:55346)
0619 02:37:12.589424 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:12.589439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:12.589518 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:12.590636 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:12.590640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:12.590659 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:12.590791 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.590798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.592405 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.592408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.593646 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.593657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.594758 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.594762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.594862 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.594932 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 02:37:12.594982 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:37:12.595071 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 02:37:12.595468 22354 tablet_bootstrap.cc:654] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:12.595913 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.589674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.7.194.195:33427)
0619 02:37:12.590063 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:12.590080 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:12.590110 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:37:12.590197 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:12.590203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:12.590456 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:12.590968 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:12.591129 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.592222 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.593790 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.594622 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.595402 (+ 780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.595611 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:12.595668 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 02:37:12.595767 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":147,"threads_started":1}
I20260619 02:37:12.596712 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.590187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:36529 (local address 127.0.0.1:41650)
0619 02:37:12.590660 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:12.590675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:12.590764 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:12.592586 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:12.592589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:12.592607 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:12.592696 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.592702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.593656 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.593660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.594818 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:12.594826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.596221 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.596224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:12.596292 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.596311 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:37:12.596407 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:37:12.596540 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":104,"threads_started":1}
I20260619 02:37:12.597157 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:12.591553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.7.194.194:36529)
0619 02:37:12.592112 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:12.592123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:12.592144 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:37:12.592216 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:12.592221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:12.592455 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:12.592609 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:12.592789 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.593478 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.594963 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:12.595912 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:12.596031 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:12.596166 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:12.596883 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 02:37:12.596993 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":132,"threads_started":1}
I20260619 02:37:12.597261 22354 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: No bootstrap required, opened a new log
I20260619 02:37:12.597352 22354 ts_tablet_manager.cc:1397] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:37:12.597843 22354 raft_consensus.cc:348] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:12.597970 22354 raft_consensus.cc:374] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:12.598028 22354 raft_consensus.cc:729] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10de6179d594e97a0b4261ef5a48c10, State: Initialized, Role: FOLLOWER
I20260619 02:37:12.598189 22354 consensus_queue.cc:260] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:12.598445 22354 ts_tablet_manager.cc:1428] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:37:12.599093 22535 tablet_service.cc:1467] Processing CreateTablet for tablet 72f3ef36eb8f4509b8afd322449835f0 (DEFAULT_TABLE table=test-table [id=55a47fe920d643348ef329c3216cb786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:12.599375 22667 tablet_service.cc:1467] Processing CreateTablet for tablet 72f3ef36eb8f4509b8afd322449835f0 (DEFAULT_TABLE table=test-table [id=55a47fe920d643348ef329c3216cb786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:12.600231 22535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f3ef36eb8f4509b8afd322449835f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:12.600262 22667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f3ef36eb8f4509b8afd322449835f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:12.604485 22493 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Bootstrap starting.
I20260619 02:37:12.606964 22493 tablet_bootstrap.cc:654] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:12.607857 22493 log.cc:824] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:12.608294 22746 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Bootstrap starting.
I20260619 02:37:12.609754 22493 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: No bootstrap required, opened a new log
I20260619 02:37:12.609953 22493 ts_tablet_manager.cc:1397] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:37:12.610698 22746 tablet_bootstrap.cc:654] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:12.611579 22746 log.cc:824] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:12.613662 22746 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: No bootstrap required, opened a new log
I20260619 02:37:12.613473 22493 raft_consensus.cc:348] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:12.613811 22493 raft_consensus.cc:374] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:12.613869 22746 ts_tablet_manager.cc:1397] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260619 02:37:12.613891 22493 raft_consensus.cc:729] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718f816bee234196a60df46408475c47, State: Initialized, Role: FOLLOWER
I20260619 02:37:12.614367 22493 consensus_queue.cc:260] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:12.615033 22601 heartbeater.cc:503] Master 127.7.194.254:38323 was elected leader, sending a full tablet report...
I20260619 02:37:12.615358 22493 ts_tablet_manager.cc:1428] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:37:12.617595 22746 raft_consensus.cc:348] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:12.617890 22746 raft_consensus.cc:374] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:12.617954 22746 raft_consensus.cc:729] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce53c8b4ce8468e94b0f7a568d5b06d, State: Initialized, Role: FOLLOWER
I20260619 02:37:12.618410 22746 consensus_queue.cc:260] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:12.619104 22733 heartbeater.cc:503] Master 127.7.194.254:38323 was elected leader, sending a full tablet report...
I20260619 02:37:12.619367 22746 ts_tablet_manager.cc:1428] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260619 02:37:12.645821 22463 tablet.cc:2369] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:37:12.798192 22734 tablet.cc:2369] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:37:12.857847 22602 tablet.cc:2369] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:37:13.030654 22468 raft_consensus.cc:482] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:37:13.030889 22468 raft_consensus.cc:504] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:13.032058 22468 leader_election.cc:288] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427), 718f816bee234196a60df46408475c47 (127.7.194.194:36529)
I20260619 02:37:13.036085 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.032429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:51685 (local address 127.7.194.194:36529)
0619 02:37:13.032593 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:13.032598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:13.032879 (+ 281us) server_negotiation.cc:407] Connection header received
0619 02:37:13.033068 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:13.033071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:13.033119 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:13.033190 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:13.033461 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.034009 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.035075 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.035480 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.035621 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:13.035674 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:13.035872 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 02:37:13.035945 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:37:13.036103 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.032333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:36529 (local address 127.7.194.193:51685)
0619 02:37:13.032772 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.032788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:13.032926 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:13.033223 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:13.033225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:13.033243 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:13.033347 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:13.033352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.034132 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.034138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:13.034933 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:13.034941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.035611 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.035615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:13.035718 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:13.035762 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:37:13.035854 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:37:13.035945 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260619 02:37:13.036301 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.032452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:49951 (local address 127.7.194.195:33427)
0619 02:37:13.032612 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:13.032619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:13.032942 (+ 323us) server_negotiation.cc:407] Connection header received
0619 02:37:13.033318 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:13.033320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:13.033361 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:13.033418 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:13.033846 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.034336 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.035326 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.035721 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.035855 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:13.035902 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:13.036104 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 02:37:13.036177 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:37:13.036319 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.032336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:33427 (local address 127.7.194.193:49951)
0619 02:37:13.032861 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.032874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:13.032948 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:13.033638 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:13.033640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:13.033657 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:13.033729 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:13.033733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.034451 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.034456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:13.035202 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:13.035211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.035848 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.035852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:13.035975 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:13.035992 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:37:13.036088 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:37:13.036155 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":203,"thread_start_us":63,"threads_started":1}
I20260619 02:37:13.036887 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f3ef36eb8f4509b8afd322449835f0" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718f816bee234196a60df46408475c47" is_pre_election: true
I20260619 02:37:13.036887 22687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f3ef36eb8f4509b8afd322449835f0" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" is_pre_election: true
I20260619 02:37:13.037235 22687 raft_consensus.cc:2454] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 0.
I20260619 02:37:13.037235 22555 raft_consensus.cc:2454] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 0.
I20260619 02:37:13.037793 22351 leader_election.cc:302] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 718f816bee234196a60df46408475c47, 7ce53c8b4ce8468e94b0f7a568d5b06d, f10de6179d594e97a0b4261ef5a48c10; no voters:
I20260619 02:37:13.038062 22468 raft_consensus.cc:2781] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:37:13.038144 22468 raft_consensus.cc:482] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:37:13.038187 22468 raft_consensus.cc:3037] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:13.039283 22468 raft_consensus.cc:504] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:13.039651 22468 leader_election.cc:288] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 1 election: Requested vote from peers 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427), 718f816bee234196a60df46408475c47 (127.7.194.194:36529)
I20260619 02:37:13.040082 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f3ef36eb8f4509b8afd322449835f0" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718f816bee234196a60df46408475c47"
I20260619 02:37:13.040068 22687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f3ef36eb8f4509b8afd322449835f0" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d"
I20260619 02:37:13.040215 22687 raft_consensus.cc:3037] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:13.040215 22555 raft_consensus.cc:3037] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:13.041529 22687 raft_consensus.cc:2454] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 1.
I20260619 02:37:13.041529 22555 raft_consensus.cc:2454] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 1.
I20260619 02:37:13.041839 22349 leader_election.cc:302] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce53c8b4ce8468e94b0f7a568d5b06d, f10de6179d594e97a0b4261ef5a48c10; no voters:
I20260619 02:37:13.042014 22468 raft_consensus.cc:2781] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:37:13.042217 22468 raft_consensus.cc:686] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 LEADER]: Becoming Leader. State: Replica: f10de6179d594e97a0b4261ef5a48c10, State: Running, Role: LEADER
I20260619 02:37:13.042503 22468 consensus_queue.cc:237] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:13.045804 22272 catalog_manager.cc:5495] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 reported cstate change: term changed from 0 to 1, leader changed from <none> to f10de6179d594e97a0b4261ef5a48c10 (127.7.194.193). New cstate: current_term: 1 leader_uuid: "f10de6179d594e97a0b4261ef5a48c10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } health_report { overall_health: HEALTHY } } }
I20260619 02:37:13.059955 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.056093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.7.194.193:37505)
0619 02:37:13.056432 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:13.056440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:13.056451 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:37:13.056524 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:13.056526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:13.056578 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:13.056709 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:37:13.056990 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.057520 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.058462 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:13.058704 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:13.058814 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:13.059092 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:37:13.059548 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:37:13.059681 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:13.059722 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 02:37:13.059776 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":125,"threads_started":1}
I20260619 02:37:13.064137 22555 raft_consensus.cc:1261] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Refusing update from remote peer f10de6179d594e97a0b4261ef5a48c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:13.064137 22687 raft_consensus.cc:1261] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 1 FOLLOWER]: Refusing update from remote peer f10de6179d594e97a0b4261ef5a48c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:13.064908 22468 consensus_queue.cc:1035] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:13.065181 22753 consensus_queue.cc:1035] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:13.072243 22756 mvcc.cc:205] Tried to move back new op lower bound from 7298402849023791104 to 7298402848944885760. Current Snapshot: MvccSnapshot[applied={T|T < 7298402849023791104}]
I20260619 02:37:13.074271 22757 mvcc.cc:205] Tried to move back new op lower bound from 7298402849023791104 to 7298402848944885760. Current Snapshot: MvccSnapshot[applied={T|T < 7298402849023791104}]
I20260619 02:37:13.081184 22273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:13.081367 22273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:13.083911 22273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Sending DeleteTablet for 3 replicas of tablet 72f3ef36eb8f4509b8afd322449835f0
I20260619 02:37:13.084648 22667 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f3ef36eb8f4509b8afd322449835f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:13 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:55346
I20260619 02:37:13.084959 22396 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f3ef36eb8f4509b8afd322449835f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:13 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:39446
I20260619 02:37:13.084944 22535 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f3ef36eb8f4509b8afd322449835f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:13 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:41650
I20260619 02:37:13.085348 22767 tablet_replica.cc:330] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: stopping tablet replica
I20260619 02:37:13.085480 22767 raft_consensus.cc:2229] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:37:13.085711 22766 tablet_replica.cc:330] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: stopping tablet replica
I20260619 02:37:13.085774 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 22311
I20260619 02:37:13.090094 22768 tablet_replica.cc:330] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: stopping tablet replica
I20260619 02:37:13.090404 22768 raft_consensus.cc:2229] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:13.090423 22766 raft_consensus.cc:2229] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:13.090752 22768 raft_consensus.cc:2258] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:13.090754 22766 raft_consensus.cc:2258] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:37:13.091768 22257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:37505 (error 108)
I20260619 02:37:13.092378 22766 ts_tablet_manager.cc:1905] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:37:13.093070 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:37:13.092446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:37505 (local address 127.0.0.1:39468)
0619 02:37:13.092622 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.092638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:13.092720 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:13.092949 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:37505: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 02:37:13.093232 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 22467
W20260619 02:37:13.093909 22257 catalog_manager.cc:4584] TS f10de6179d594e97a0b4261ef5a48c10 (127.7.194.193:37505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72f3ef36eb8f4509b8afd322449835f0: Network error: Client connection negotiation failed: client connection to 127.7.194.193:37505: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:37:13.095566 22766 ts_tablet_manager.cc:1918] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:13.095676 22766 log.cc:1196] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/72f3ef36eb8f4509b8afd322449835f0
I20260619 02:37:13.096021 22766 ts_tablet_manager.cc:1939] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Deleting consensus metadata
I20260619 02:37:13.097374 22258 catalog_manager.cc:4841] TS 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427): tablet 72f3ef36eb8f4509b8afd322449835f0 (table test-table [id=55a47fe920d643348ef329c3216cb786]) successfully deleted
I20260619 02:37:13.099224 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 22604
I20260619 02:37:13.100281 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.099463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:36529 (local address 127.0.0.1:41654)
0619 02:37:13.100131 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.100204 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.194:36529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591}
I20260619 02:37:13.106037 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 22240
I20260619 02:37:13.112645 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38323
--webserver_interface=127.7.194.254
--webserver_port=36055
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:13.225180 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:13.225466 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:13.225507 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:13.230240 22770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:13.230306 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:13.230327 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:13.230345 22770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:13.230386 22770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:13.235867 22770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=36055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:13.237095 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:13.238233 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:13.247071 22778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:13.246990 22776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:13.247006 22775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:13.247377 22770 server_base.cc:1034] running on GCE node
I20260619 02:37:13.247961 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:13.249074 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:13.250231 22770 hybrid_clock.cc:648] HybridClock initialized: now 1781836633250193 us; error 51 us; skew 500 ppm
Jun 19 02:37:13 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836633, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:13.253089 22770 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:37:13.254199 22770 webserver.cc:460] Webserver started at http://127.7.194.254:36055/ using document root <none> and password file <none>
I20260619 02:37:13.254781 22770 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:13.254844 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:13.258791 22770 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:37:13.261089 22785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:13.262423 22770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:37:13.262555 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "0c43ede0cbc74c7282b537f93aca3a16"
format_stamp: "Formatted at 2026-06-19 02:37:10 on dist-test-slave-9tkz"
server_key: "9d388a9fbfacacb60ff139db697b3319"
server_key_iv: "00eca4dd220c99dad37fefe77213c2b0"
server_key_version: "encryptionkey@0"
I20260619 02:37:13.263161 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:13.277567 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:13.280755 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:13.280959 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:13.288259 22770 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:38323
I20260619 02:37:13.288645 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:38323 every 8 connection(s)
I20260619 02:37:13.289477 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:37:13.294500 22838 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:37:13.296257 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Bootstrap starting.
I20260619 02:37:13.299521 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 22770
I20260619 02:37:13.300006 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:37505
--local_ip_for_outbound_sockets=127.7.194.193
--tserver_master_addrs=127.7.194.254:38323
--webserver_port=41307
--webserver_interface=127.7.194.193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:13.314720 22838 log.cc:824] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:13.326128 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:37:13.326514 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16: Bootstrap complete.
I20260619 02:37:13.330663 22838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:13.331054 22838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c43ede0cbc74c7282b537f93aca3a16, State: Initialized, Role: FOLLOWER
I20260619 02:37:13.331550 22838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:13.331662 22838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:37:13.331750 22838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:37:13.331849 22838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:37:13.333791 22838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:13.334138 22838 leader_election.cc:302] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c43ede0cbc74c7282b537f93aca3a16; no voters:
I20260619 02:37:13.334534 22838 leader_election.cc:288] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:37:13.334625 22842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:37:13.334810 22842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [term 3 LEADER]: Becoming Leader. State: Replica: 0c43ede0cbc74c7282b537f93aca3a16, State: Running, Role: LEADER
I20260619 02:37:13.335103 22842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } }
I20260619 02:37:13.335773 22838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:37:13.336052 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c43ede0cbc74c7282b537f93aca3a16. Latest consensus state: current_term: 3 leader_uuid: "0c43ede0cbc74c7282b537f93aca3a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } } }
I20260619 02:37:13.336161 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:13.336509 22846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:37:13.336607 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0c43ede0cbc74c7282b537f93aca3a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43ede0cbc74c7282b537f93aca3a16" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38323 } } }
I20260619 02:37:13.336782 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c43ede0cbc74c7282b537f93aca3a16 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:13.341917 22846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:37:13.342612 22846 catalog_manager.cc:1259] Loaded cluster ID: 460c1183fd8a4a5f919287840be21755
I20260619 02:37:13.342693 22846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 02:37:13.343489 22846 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fef6c0849ad google::LogMessage::Fail() at ??:0
@ 0x7fef6c08788a google::LogMessage::SendToLog() at ??:0
@ 0x7fef6c0844de google::LogMessage::Flush() at ??:0
@ 0x7fef6c085a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fef769c3fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fef769c4d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fef769c1579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fef76a05640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fef6e2d63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fef6e2d6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fef6e2d83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fef6e2c7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fef6f90b6db start_thread at ??:0
@ 0x7fef6ab8261f clone at ??:0
W20260619 02:37:13.430178 22840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:13.430487 22840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:13.430600 22840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:13.435479 22840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:13.435581 22840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:13.435686 22840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:37:13.441375 22840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:37505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=41307
--tserver_master_addrs=127.7.194.254:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:13.442641 22840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:13.443753 22840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:13.453728 22882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:13.453773 22880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:13.454073 22840 server_base.cc:1034] running on GCE node
W20260619 02:37:13.453747 22879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:13.454674 22840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:13.455476 22840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:13.456718 22840 hybrid_clock.cc:648] HybridClock initialized: now 1781836633456673 us; error 42 us; skew 500 ppm
Jun 19 02:37:13 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836633, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:13.460556 22840 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:37:13.461791 22840 webserver.cc:460] Webserver started at http://127.7.194.193:41307/ using document root <none> and password file <none>
I20260619 02:37:13.462616 22840 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:13.462720 22840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:13.468227 22840 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 02:37:13.471226 22889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:13.473023 22840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:37:13.473172 22840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "f10de6179d594e97a0b4261ef5a48c10"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "dee9f4e20a120e58dcaa9736c7e0d9ca"
server_key_iv: "c7bfc179e92ac6cf6a4a635cf1c2eeaa"
server_key_version: "encryptionkey@0"
I20260619 02:37:13.474304 22840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:13.515786 22840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:13.520098 22840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:13.520337 22840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:13.521102 22840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:13.522423 22896 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:37:13.528311 22840 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:37:13.528424 22840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 02:37:13.528528 22840 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:37:13.530393 22840 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:37:13.530471 22840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 02:37:13.530794 22896 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Bootstrap starting.
I20260619 02:37:13.541359 22840 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:37505
I20260619 02:37:13.541666 23003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:37505 every 8 connection(s)
I20260619 02:37:13.542531 22840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:37:13.547832 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 22840
I20260619 02:37:13.548339 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:36529
--local_ip_for_outbound_sockets=127.7.194.194
--tserver_master_addrs=127.7.194.254:38323
--webserver_port=45993
--webserver_interface=127.7.194.194
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:13.554543 22896 log.cc:824] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:13.565261 22896 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:37:13.565785 22896 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Bootstrap complete.
I20260619 02:37:13.567274 22896 ts_tablet_manager.cc:1397] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.012s
I20260619 02:37:13.570758 22896 raft_consensus.cc:348] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:13.571331 22896 raft_consensus.cc:729] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10de6179d594e97a0b4261ef5a48c10, State: Initialized, Role: FOLLOWER
I20260619 02:37:13.572009 22896 consensus_queue.cc:260] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:13.573181 22896 ts_tablet_manager.cc:1428] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260619 02:37:13.608676 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:37:13.544607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:43323)
0619 02:37:13.545129 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.545187 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:13.546081 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:13.607866 (+ 61785us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":155,"threads_started":1}
W20260619 02:37:13.609182 23004 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:38323 (0 consecutive failures): Network error: Failed to ping master at 127.7.194.254:38323: Client connection negotiation failed: client connection to 127.7.194.254:38323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:37:13.610013 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.609741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:46641)
0619 02:37:13.609876 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.609953 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 02:37:13.610718 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.610516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:34775)
0619 02:37:13.610626 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.610669 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 02:37:13.610955 23004 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:37:13.670950 23007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:13.671236 23007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:13.671314 23007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:13.675495 23007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:13.675562 23007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:13.675660 23007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:37:13.681136 23007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:36529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=45993
--tserver_master_addrs=127.7.194.254:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:13.682355 23007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:13.683400 23007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:13.693480 23015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:13.693564 23007 server_base.cc:1034] running on GCE node
W20260619 02:37:13.693516 23017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:13.693516 23014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:13.694154 23007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:13.694823 23007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:13.696025 23007 hybrid_clock.cc:648] HybridClock initialized: now 1781836633696007 us; error 33 us; skew 500 ppm
Jun 19 02:37:13 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836633, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:13.698899 23007 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:37:13.699805 23007 webserver.cc:460] Webserver started at http://127.7.194.194:45993/ using document root <none> and password file <none>
I20260619 02:37:13.700394 23007 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:13.700489 23007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:13.704476 23007 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:37:13.706743 23024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:13.708057 23007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:37:13.708204 23007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "718f816bee234196a60df46408475c47"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "10e1266cf5c0199b340cdf83dd2dc64a"
server_key_iv: "e3bcf540b46e583f8a458b4baf552dd6"
server_key_version: "encryptionkey@0"
I20260619 02:37:13.708765 23007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:13.739303 23007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:13.742595 23007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:13.742820 23007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:13.743466 23007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:13.744590 23031 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:37:13.748936 23007 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:37:13.749027 23007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 02:37:13.749112 23007 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:37:13.750952 23007 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:37:13.751026 23007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 02:37:13.751349 23031 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Bootstrap starting.
I20260619 02:37:13.757236 23031 log.cc:824] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:13.761715 23007 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:36529
I20260619 02:37:13.761777 23139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:36529 every 8 connection(s)
I20260619 02:37:13.762604 23007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:37:13.765591 23031 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:37:13.766012 23031 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Bootstrap complete.
I20260619 02:37:13.766012 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.764474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:34861)
0619 02:37:13.764966 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.765102 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
I20260619 02:37:13.766399 23031 ts_tablet_manager.cc:1397] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
W20260619 02:37:13.766460 23140 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:38323 (0 consecutive failures): Network error: Failed to ping master at 127.7.194.254:38323: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
I20260619 02:37:13.767068 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.766833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:53341)
0619 02:37:13.766951 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.767007 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:37:13.767799 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.767539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:55107)
0619 02:37:13.767678 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.767733 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:37:13.767874 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 23007
I20260619 02:37:13.768271 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:33427
--local_ip_for_outbound_sockets=127.7.194.195
--tserver_master_addrs=127.7.194.254:38323
--webserver_port=37329
--webserver_interface=127.7.194.195
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:13.769515 23140 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:37:13.769767 23031 raft_consensus.cc:348] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:13.770412 23031 raft_consensus.cc:729] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718f816bee234196a60df46408475c47, State: Initialized, Role: FOLLOWER
I20260619 02:37:13.770928 23031 consensus_queue.cc:260] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:13.771858 23031 ts_tablet_manager.cc:1428] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:37:13.771970 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.771695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:47941)
0619 02:37:13.771809 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.771861 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:37:13.772569 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.772382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:50175)
0619 02:37:13.772474 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.772514 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:37:13.882972 23143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:13.883318 23143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:13.883414 23143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:13.887778 23143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:13.887869 23143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:13.887959 23143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:37:13.893724 23143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:33427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=37329
--tserver_master_addrs=127.7.194.254:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:13.894907 23143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:13.896009 23143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:13.906124 23149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:13.906200 23152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:13.906165 23150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:13.906622 23143 server_base.cc:1034] running on GCE node
I20260619 02:37:13.907045 23143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:13.907788 23143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:13.908978 23143 hybrid_clock.cc:648] HybridClock initialized: now 1781836633908958 us; error 35 us; skew 500 ppm
Jun 19 02:37:13 dist-test-slave-9tkz krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836633, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:13.911891 23143 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:37:13.912963 23143 webserver.cc:460] Webserver started at http://127.7.194.195:37329/ using document root <none> and password file <none>
I20260619 02:37:13.913729 23143 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:13.913834 23143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:13.917912 23143 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:37:13.920094 23159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:13.921427 23143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:37:13.921587 23143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d"
format_stamp: "Formatted at 2026-06-19 02:37:11 on dist-test-slave-9tkz"
server_key: "b84820a38e1cd5805af5e2cc13de3f49"
server_key_iv: "df137df1c15f360739223fd1e3e0931e"
server_key_version: "encryptionkey@0"
I20260619 02:37:13.922214 23143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:13.941123 23143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:13.944617 23143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:13.944868 23143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:13.945679 23143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:13.946592 23143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:13.946671 23143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:13.946743 23143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:13.946787 23143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:13.956188 23143 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:33427
I20260619 02:37:13.956210 23272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:33427 every 8 connection(s)
I20260619 02:37:13.957288 23143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:37:13.961477 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.959623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:59945)
0619 02:37:13.960089 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.960212 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
W20260619 02:37:13.962105 23273 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:38323 (0 consecutive failures): Network error: Failed to ping master at 127.7.194.254:38323: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
I20260619 02:37:13.962715 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.962488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:44641)
0619 02:37:13.962594 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.962657 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 02:37:13.963358 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:13.963186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:59929)
0619 02:37:13.963275 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:13.963314 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 02:37:13.963536 23273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:37:13.966272 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 23143
I20260619 02:37:14.612067 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:14.611615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:33893)
0619 02:37:14.611865 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:14.611925 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260619 02:37:14.773800 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:14.773338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:46325)
0619 02:37:14.773648 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:14.773710 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260619 02:37:14.964545 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:14.964134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:34719)
0619 02:37:14.964383 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:14.964441 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260619 02:37:15.346971 23280 raft_consensus.cc:482] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:37:15.347167 23280 raft_consensus.cc:504] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:15.348279 23280 leader_election.cc:288] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427), 718f816bee234196a60df46408475c47 (127.7.194.194:36529)
Jun 19 02:37:15 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836633, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.195@KRBTEST.COM
Jun 19 02:37:15 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836633, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.194@KRBTEST.COM
I20260619 02:37:15.362126 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.348469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:33427 (local address 127.7.194.193:33861)
0619 02:37:15.348788 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:15.348803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:15.348910 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:15.350328 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:15.350331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:15.350805 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:15.351315 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:15.351330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:15.352609 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:15.352615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:15.353658 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:15.353667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:15.354513 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:15.354517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:15.354762 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:15.355435 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:15.355459 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:15.357073 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:15.359216 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:15.359223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:15.359237 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:15.359580 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:15.360033 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:15.360038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:15.360040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:15.360244 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:15.361020 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:15.361027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:15.361346 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 02:37:15.361588 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 02:37:15.361925 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":139,"spinlock_wait_cycles":25728,"thread_start_us":79,"threads_started":1}
I20260619 02:37:15.362341 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.348620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:33861 (local address 127.7.194.195:33427)
0619 02:37:15.348934 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:15.348974 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:15.349023 (+ 49us) server_negotiation.cc:407] Connection header received
0619 02:37:15.349791 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:15.349796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:15.350112 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:15.350378 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:15.351481 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:15.352425 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:15.353796 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:15.354390 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:15.354566 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:15.357354 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:15.357361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:15.357378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:15.357415 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:15.359114 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:15.359744 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:15.359749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:15.359753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:15.359921 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:15.360343 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:15.360350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:15.360355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:15.360892 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:15.361054 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:15.361792 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 02:37:15.362153 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260619 02:37:15.362553 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.348785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:39119 (local address 127.7.194.194:36529)
0619 02:37:15.349230 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:15.349241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:15.349272 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:37:15.349450 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:15.349454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:15.349862 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:15.350187 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:15.351493 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:15.352448 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:15.353793 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:15.354387 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:15.354562 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:15.357718 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:15.357726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:15.357743 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:15.357776 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:15.360112 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:15.360650 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:15.360656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:15.360660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:15.360770 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:15.361075 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:15.361078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:15.361079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:15.361571 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:15.361703 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:15.362071 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 02:37:15.362379 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":78,"threads_started":1}
I20260619 02:37:15.362327 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.348548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:36529 (local address 127.7.194.193:39119)
0619 02:37:15.349022 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:15.349037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:15.349116 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:15.350224 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:15.350231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:15.350805 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:15.351315 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:15.351331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:15.352581 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:15.352600 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:15.353674 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:15.353684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:15.354513 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:15.354517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:15.354769 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:15.355498 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:15.355510 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:15.357519 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:15.360255 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:15.360257 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:15.360258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:15.360540 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:15.360880 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:15.360883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:15.360884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:15.360984 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:15.361690 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:15.361694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:15.361797 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 02:37:15.361932 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 02:37:15.362081 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":163,"spinlock_wait_cycles":8448,"thread_start_us":56,"threads_started":1}
I20260619 02:37:15.363970 23093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f3ef36eb8f4509b8afd322449835f0" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "718f816bee234196a60df46408475c47" is_pre_election: true
I20260619 02:37:15.364310 23093 raft_consensus.cc:2454] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 1.
I20260619 02:37:15.365162 22893 leader_election.cc:302] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718f816bee234196a60df46408475c47, f10de6179d594e97a0b4261ef5a48c10; no voters:
W20260619 02:37:15.365351 22891 leader_election.cc:341] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427): Not found: Tablet not found: 72f3ef36eb8f4509b8afd322449835f0
I20260619 02:37:15.365440 23280 raft_consensus.cc:2781] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:37:15.365599 23280 raft_consensus.cc:482] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:37:15.365659 23280 raft_consensus.cc:3037] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:37:15.367743 23280 raft_consensus.cc:504] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:15.368104 23280 leader_election.cc:288] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 2 election: Requested vote from peers 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427), 718f816bee234196a60df46408475c47 (127.7.194.194:36529)
I20260619 02:37:15.368513 23093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f3ef36eb8f4509b8afd322449835f0" candidate_uuid: "f10de6179d594e97a0b4261ef5a48c10" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "718f816bee234196a60df46408475c47"
I20260619 02:37:15.368646 23093 raft_consensus.cc:3037] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 1 FOLLOWER]: Advancing to term 2
W20260619 02:37:15.368721 22891 leader_election.cc:341] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427): Not found: Tablet not found: 72f3ef36eb8f4509b8afd322449835f0
I20260619 02:37:15.370836 23093 raft_consensus.cc:2454] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f10de6179d594e97a0b4261ef5a48c10 in term 2.
I20260619 02:37:15.371179 22893 leader_election.cc:302] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 718f816bee234196a60df46408475c47, f10de6179d594e97a0b4261ef5a48c10; no voters: 7ce53c8b4ce8468e94b0f7a568d5b06d
I20260619 02:37:15.371377 23280 raft_consensus.cc:2781] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:37:15.371614 23280 raft_consensus.cc:686] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [term 2 LEADER]: Becoming Leader. State: Replica: f10de6179d594e97a0b4261ef5a48c10, State: Running, Role: LEADER
I20260619 02:37:15.371963 23280 consensus_queue.cc:237] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:15.372285 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.372044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:35627)
0619 02:37:15.372160 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:15.372212 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:37:15.775141 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.774586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:39819)
0619 02:37:15.774936 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:15.775019 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":94,"threads_started":1}
W20260619 02:37:15.883692 22891 consensus_peers.cc:487] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 -> Peer 7ce53c8b4ce8468e94b0f7a568d5b06d (127.7.194.195:33427): Couldn't send request to peer 7ce53c8b4ce8468e94b0f7a568d5b06d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72f3ef36eb8f4509b8afd322449835f0. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:37:15.948382 23093 raft_consensus.cc:1261] T 72f3ef36eb8f4509b8afd322449835f0 P 718f816bee234196a60df46408475c47 [term 2 FOLLOWER]: Refusing update from remote peer f10de6179d594e97a0b4261ef5a48c10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 02:37:15.949178 23288 consensus_queue.cc:1035] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 02:37:15.951555 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.951237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:52819)
0619 02:37:15.951368 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:15.951429 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 02:37:15.954444 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.954072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:46129)
0619 02:37:15.954263 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:15.954313 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":53,"threads_started":1}
I20260619 02:37:15.954865 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.954622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:60663)
0619 02:37:15.954737 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:15.954793 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:37:15.965880 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:15.965307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:59151)
0619 02:37:15.965652 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:15.965738 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260619 02:37:16.339979 23296 ts_tablet_manager.cc:927] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Initiating tablet copy from peer f10de6179d594e97a0b4261ef5a48c10 (127.7.194.193:37505)
I20260619 02:37:16.340610 23296 tablet_copy_client.cc:316] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.194.193:37505
Jun 19 02:37:16 dist-test-slave-9tkz krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836633, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:37:16.351480 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:16.341315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:39541 (local address 127.7.194.193:37505)
0619 02:37:16.341711 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:16.341721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:16.341743 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:37:16.341791 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:16.341796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:16.341981 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:16.342108 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:16.342778 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:16.343531 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:16.344379 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:16.344730 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:16.344808 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:16.347340 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:16.347349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:16.347361 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:16.347390 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:16.349264 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:16.349919 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:16.349925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:16.349927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:16.350035 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:16.350343 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:16.350348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:16.350350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:16.350630 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:16.350781 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:16.351164 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 02:37:16.351322 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":92,"threads_started":1}
I20260619 02:37:16.351495 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:16.341104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:37505 (local address 127.7.194.195:39541)
0619 02:37:16.341257 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:16.341278 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:16.341411 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:16.342145 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:16.342151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:16.342544 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:16.342663 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:16.342669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:16.343638 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:16.343641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:16.344279 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:16.344287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:16.344826 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:16.344828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:16.344907 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:16.345521 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:16.345533 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:16.347158 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:16.349399 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:16.349410 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:16.349417 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:16.349775 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:16.350142 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:16.350146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:16.350148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:16.350245 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:16.350759 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:16.350766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:16.350906 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 02:37:16.351074 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 02:37:16.351284 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 02:37:16.352682 22978 tablet_copy_service.cc:140] P f10de6179d594e97a0b4261ef5a48c10: Received BeginTabletCopySession request for tablet 72f3ef36eb8f4509b8afd322449835f0 from peer 7ce53c8b4ce8468e94b0f7a568d5b06d ({username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:39541)
I20260619 02:37:16.352846 22978 tablet_copy_service.cc:161] P f10de6179d594e97a0b4261ef5a48c10: Beginning new tablet copy session on tablet 72f3ef36eb8f4509b8afd322449835f0 from peer 7ce53c8b4ce8468e94b0f7a568d5b06d at {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:39541: session id = 7ce53c8b4ce8468e94b0f7a568d5b06d-72f3ef36eb8f4509b8afd322449835f0
I20260619 02:37:16.354178 22978 tablet_copy_source_session.cc:205] T 72f3ef36eb8f4509b8afd322449835f0 P f10de6179d594e97a0b4261ef5a48c10: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:37:16.358451 23296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f3ef36eb8f4509b8afd322449835f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:16.366467 23296 tablet_copy_client.cc:791] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: tablet copy: Starting download of 0 data blocks...
I20260619 02:37:16.366671 23296 tablet_copy_client.cc:655] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: tablet copy: Starting download of 1 WAL segments...
I20260619 02:37:16.368465 23296 tablet_copy_client.cc:531] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:37:16.370373 23296 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Bootstrap starting.
I20260619 02:37:16.375720 23296 log.cc:824] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:16.382867 23296 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:37:16.383149 23296 tablet_bootstrap.cc:492] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Bootstrap complete.
I20260619 02:37:16.383428 23296 ts_tablet_manager.cc:1397] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260619 02:37:16.385787 23296 raft_consensus.cc:348] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:16.386044 23296 raft_consensus.cc:729] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce53c8b4ce8468e94b0f7a568d5b06d, State: Initialized, Role: FOLLOWER
I20260619 02:37:16.386422 23296 consensus_queue.cc:260] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce53c8b4ce8468e94b0f7a568d5b06d" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 33427 } } peers { permanent_uuid: "718f816bee234196a60df46408475c47" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 36529 } } peers { permanent_uuid: "f10de6179d594e97a0b4261ef5a48c10" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 37505 } }
I20260619 02:37:16.387349 23296 ts_tablet_manager.cc:1428] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 02:37:16.387639 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:16.387359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:42127)
0619 02:37:16.387515 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:16.387570 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:37:16.388017 22978 tablet_copy_service.cc:342] P f10de6179d594e97a0b4261ef5a48c10: Request end of tablet copy session 7ce53c8b4ce8468e94b0f7a568d5b06d-72f3ef36eb8f4509b8afd322449835f0 received from {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:39541
I20260619 02:37:16.388171 22978 tablet_copy_service.cc:433] P f10de6179d594e97a0b4261ef5a48c10: ending tablet copy session 7ce53c8b4ce8468e94b0f7a568d5b06d-72f3ef36eb8f4509b8afd322449835f0 on tablet 72f3ef36eb8f4509b8afd322449835f0 with peer 7ce53c8b4ce8468e94b0f7a568d5b06d
I20260619 02:37:16.388386 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:16.388139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:40701)
0619 02:37:16.388290 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:16.388339 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 02:37:16.874688 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:16.874315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:51303)
0619 02:37:16.874542 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:16.874593 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":45,"threads_started":1}
I20260619 02:37:16.874727 23227 raft_consensus.cc:1203] T 72f3ef36eb8f4509b8afd322449835f0 P 7ce53c8b4ce8468e94b0f7a568d5b06d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 02:37:16.875756 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:16.875441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:57825)
0619 02:37:16.875571 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:16.875679 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:37:16.956019 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:16.955567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:54915)
0619 02:37:16.955838 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:16.955892 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":70,"threads_started":1}
I20260619 02:37:17.875895 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:17.875475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:34543)
0619 02:37:17.875738 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:17.875798 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":77,"threads_started":1}
I20260619 02:37:17.876833 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:17.876427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:40155)
0619 02:37:17.876702 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:17.876755 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260619 02:37:17.957489 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:17.956972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:38553)
0619 02:37:17.957301 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:17.957380 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":86,"threads_started":1}
I20260619 02:37:18.877470 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:18.876900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:33843)
0619 02:37:18.877259 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:18.877333 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260619 02:37:18.878160 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:18.877679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:57379)
0619 02:37:18.878018 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:18.878073 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":148,"threads_started":1}
I20260619 02:37:18.878394 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:18.878123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:40131)
0619 02:37:18.878242 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:18.878302 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:37:18.959045 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:18.958559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:33129)
0619 02:37:18.958847 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:18.958923 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260619 02:37:19.879586 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:19.879156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:49001)
0619 02:37:19.879426 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:19.879487 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
I20260619 02:37:19.879616 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:19.879154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:57207)
0619 02:37:19.879484 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:19.879533 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
I20260619 02:37:19.960347 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:19.959899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:42769)
0619 02:37:19.960188 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:19.960251 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":117,"threads_started":1}
I20260619 02:37:20.880967 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:20.880527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:36143)
0619 02:37:20.880787 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:20.880840 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
I20260619 02:37:20.880985 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:20.880512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:58425)
0619 02:37:20.880805 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:20.880880 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
I20260619 02:37:20.961747 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:20.961263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:37809)
0619 02:37:20.961523 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:20.961620 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260619 02:37:21.882427 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:21.881938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:51965)
0619 02:37:21.882234 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:21.882310 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
I20260619 02:37:21.882427 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:21.881929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:46019)
0619 02:37:21.882234 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:21.882310 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
I20260619 02:37:21.962976 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:21.962486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:53573)
0619 02:37:21.962772 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:21.962874 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
I20260619 02:37:22.883642 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:22.883168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:58789)
0619 02:37:22.883465 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:22.883532 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260619 02:37:22.883733 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:22.883331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:53871)
0619 02:37:22.883568 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:22.883621 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260619 02:37:22.964397 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:22.963975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:37739)
0619 02:37:22.964240 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:22.964301 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260619 02:37:23.884897 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:23.884393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:57361)
0619 02:37:23.884735 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:23.884793 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260619 02:37:23.884943 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:23.884522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:52373)
0619 02:37:23.884794 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:23.884853 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":62,"threads_started":1}
I20260619 02:37:23.965826 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:23.965320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:58121)
0619 02:37:23.965666 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:23.965727 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260619 02:37:24.886169 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:24.885731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:51497)
0619 02:37:24.886014 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:24.886070 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260619 02:37:24.886234 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:24.885850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:39137)
0619 02:37:24.886080 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:24.886142 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260619 02:37:24.967090 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:24.966652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:49361)
0619 02:37:24.966912 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:24.966974 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260619 02:37:25.887445 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:25.887032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:59379)
0619 02:37:25.887301 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:25.887357 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":105,"threads_started":1}
I20260619 02:37:25.887445 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:25.886975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:54183)
0619 02:37:25.887266 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:25.887348 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260619 02:37:25.968420 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:25.967921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:48931)
0619 02:37:25.968203 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:25.968272 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260619 02:37:26.888701 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:26.888275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:39605)
0619 02:37:26.888548 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:26.888606 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":105,"threads_started":1}
I20260619 02:37:26.888701 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:26.888299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:34359)
0619 02:37:26.888555 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:26.888607 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260619 02:37:26.969776 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:26.969243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:60257)
0619 02:37:26.969595 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:26.969664 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":90,"threads_started":1}
I20260619 02:37:27.890054 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:27.889612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:37341)
0619 02:37:27.889883 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:27.889951 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
I20260619 02:37:27.890081 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:27.889630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:45619)
0619 02:37:27.889942 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:27.889991 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":82,"threads_started":1}
I20260619 02:37:27.970943 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:27.970547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:50957)
0619 02:37:27.970793 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:27.970847 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260619 02:37:28.891433 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:28.890924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.195:49151)
0619 02:37:28.891206 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:28.891301 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260619 02:37:28.891518 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:28.891138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.193:34723)
0619 02:37:28.891387 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:28.891436 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260619 02:37:28.972499 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:28.972057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38323 (local address 127.7.194.194:39649)
0619 02:37:28.972331 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:28.972398 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260619 02:37:28.975204 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 22840
I20260619 02:37:28.981302 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 23007
I20260619 02:37:28.987105 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 23143
2026-06-19T02:37:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19549 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:37:29 dist-test-slave-9tkz krb5kdc[23346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:37:29 dist-test-slave-9tkz krb5kdc[23346](info): set up 2 sockets
Jun 19 02:37:29 dist-test-slave-9tkz krb5kdc[23346](info): commencing operation
krb5kdc: starting...
W20260619 02:37:30.038081 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:37:30 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:37:30.162765 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38615
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:30.276952 23359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:30.277253 23359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:30.277297 23359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:30.281638 23359 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:37:30.281704 23359 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:37:30.281720 23359 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:37:30.281735 23359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:30.281800 23359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:30.281833 23359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:30.281894 23359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:30.281953 23359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:30.287433 23359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:30.288654 23359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:30.289749 23359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:37:30.298669 23359 server_base.cc:1034] running on GCE node
W20260619 02:37:30.298628 23364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:30.298622 23365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:30.298629 23367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:30.299535 23359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:30.300650 23359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:30.301831 23359 hybrid_clock.cc:648] HybridClock initialized: now 1781836650301802 us; error 43 us; skew 500 ppm
Jun 19 02:37:30 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836650, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:30.304605 23359 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:37:30.305591 23359 webserver.cc:460] Webserver started at http://127.7.194.254:32927/ using document root <none> and password file <none>
I20260619 02:37:30.306180 23359 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:30.306274 23359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:30.306514 23359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:30.308676 23359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "7154e3ae52f34384900e9d44943ea048"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "eb9d5850581a0bf12c41a38c5a536f87"
server_key_iv: "1a7c069e6848430b6923e0e7b2b89944"
server_key_version: "encryptionkey@0"
I20260619 02:37:30.309274 23359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "7154e3ae52f34384900e9d44943ea048"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "eb9d5850581a0bf12c41a38c5a536f87"
server_key_iv: "1a7c069e6848430b6923e0e7b2b89944"
server_key_version: "encryptionkey@0"
I20260619 02:37:30.313074 23359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:37:30.315667 23374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:30.317054 23359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:37:30.317231 23359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "7154e3ae52f34384900e9d44943ea048"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "eb9d5850581a0bf12c41a38c5a536f87"
server_key_iv: "1a7c069e6848430b6923e0e7b2b89944"
server_key_version: "encryptionkey@0"
I20260619 02:37:30.317370 23359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:30.329120 23359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:30.332630 23359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:30.332872 23359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:30.340454 23359 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:38615
I20260619 02:37:30.340766 23426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:38615 every 8 connection(s)
I20260619 02:37:30.341538 23359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:37:30.344512 23427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:30.349309 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 23359
I20260619 02:37:30.349470 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:37:30.349848 7947 external_mini_cluster.cc:1428] Setting key c1b7727a723021db066b89a6707945ad
I20260619 02:37:30.350467 23427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Bootstrap starting.
I20260619 02:37:30.352897 23427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:30.353857 23427 log.cc:824] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:30.356145 23427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: No bootstrap required, opened a new log
Jun 19 02:37:30 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:30.360646 23427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:37:30.360913 23427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:30.361001 23427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7154e3ae52f34384900e9d44943ea048, State: Initialized, Role: FOLLOWER
I20260619 02:37:30.361534 23427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:37:30.361756 23427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:37:30.361833 23427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:37:30.361941 23427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:30.362843 23427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:37:30.363207 23427 leader_election.cc:302] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7154e3ae52f34384900e9d44943ea048; no voters:
I20260619 02:37:30.363559 23427 leader_election.cc:288] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:37:30.363638 23432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:37:30.363843 23432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 1 LEADER]: Becoming Leader. State: Replica: 7154e3ae52f34384900e9d44943ea048, State: Running, Role: LEADER
I20260619 02:37:30.364194 23432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:37:30.364934 23427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:37:30.365479 23433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7154e3ae52f34384900e9d44943ea048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } } }
I20260619 02:37:30.365623 23433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:30.365626 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:30.351436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.7.194.254:38615)
0619 02:37:30.351899 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:30.351921 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:30.351972 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:37:30.352647 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:30.352666 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:30.353080 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:30.353398 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:30.354389 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.355579 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.356770 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.357241 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.357389 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:30.360200 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:30.360206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:30.360220 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:30.360248 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:30.362311 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:30.362744 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:30.362750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:30.362756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:30.362884 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:30.363164 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:30.363169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:30.363170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:30.363661 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:30.363922 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:30.364234 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 02:37:30.364566 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":165,"threads_started":1}
I20260619 02:37:30.366323 23434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7154e3ae52f34384900e9d44943ea048. Latest consensus state: current_term: 1 leader_uuid: "7154e3ae52f34384900e9d44943ea048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } } }
I20260619 02:37:30.366426 23434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: This master's current role is: LEADER
W20260619 02:37:30.369994 23447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:37:30.370082 23447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:37:30.370152 23437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:37:30.371085 23437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:37:30.376446 23437 catalog_manager.cc:1347] Generated new cluster ID: 88a46d67c68e4c288db67f48a703f80b
I20260619 02:37:30.376523 23437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:37:30.383200 23437 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:37:30.384191 23437 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:37:30.391235 23437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Generated new TSK 0
I20260619 02:37:30.392027 23437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:37:30.424569 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38615
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:30.540508 23453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:30.540792 23453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:30.540840 23453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:30.545020 23453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:30.545094 23453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:30.545195 23453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:37:30.550761 23453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:30.551960 23453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:30.553074 23453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:30.563167 23458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:30.563180 23461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:30.563169 23459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:30.563618 23453 server_base.cc:1034] running on GCE node
I20260619 02:37:30.564064 23453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:30.564733 23453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:30.565932 23453 hybrid_clock.cc:648] HybridClock initialized: now 1781836650565912 us; error 34 us; skew 500 ppm
Jun 19 02:37:30 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836650, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:30.569126 23453 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:37:30.570154 23453 webserver.cc:460] Webserver started at http://127.7.194.193:43683/ using document root <none> and password file <none>
I20260619 02:37:30.570762 23453 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:30.570823 23453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:30.571058 23453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:30.573184 23453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "e7dae38bb20144c8af479b4a960655ae"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "f60eb4e8ccc1ba4a0ab74b811c6bb4b4"
server_key_iv: "77a7d7f2108cb6bd2678ce4d5b7269a1"
server_key_version: "encryptionkey@0"
I20260619 02:37:30.573767 23453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "e7dae38bb20144c8af479b4a960655ae"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "f60eb4e8ccc1ba4a0ab74b811c6bb4b4"
server_key_iv: "77a7d7f2108cb6bd2678ce4d5b7269a1"
server_key_version: "encryptionkey@0"
I20260619 02:37:30.577713 23453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:37:30.580271 23468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:30.581892 23453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:37:30.582031 23453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "e7dae38bb20144c8af479b4a960655ae"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "f60eb4e8ccc1ba4a0ab74b811c6bb4b4"
server_key_iv: "77a7d7f2108cb6bd2678ce4d5b7269a1"
server_key_version: "encryptionkey@0"
I20260619 02:37:30.582157 23453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:30.600337 23453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:30.603963 23453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:30.604176 23453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:30.604817 23453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:30.605743 23453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:30.605793 23453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:30.605857 23453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:30.605895 23453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:30.615321 23453 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:38627
I20260619 02:37:30.615353 23581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:38627 every 8 connection(s)
I20260619 02:37:30.616436 23453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:37:30.621474 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 23453
I20260619 02:37:30.621644 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:37:30.622035 7947 external_mini_cluster.cc:1428] Setting key dc249ec2e6eb9060209d61ab36419e9e
Jun 19 02:37:30 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836650, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:30.631539 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:30.618404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:44905 (local address 127.7.194.254:38615)
0619 02:37:30.618549 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:30.618552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:30.619414 (+ 862us) server_negotiation.cc:407] Connection header received
0619 02:37:30.620819 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:30.620825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:30.620883 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:30.620958 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:30.622387 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.623181 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.624490 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.624731 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.624844 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:30.627586 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:30.627589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:30.627591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:30.627615 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:30.629226 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:30.629903 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:30.629906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:30.629907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:30.629978 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:30.630363 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:30.630365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:30.630367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:30.630538 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:30.630609 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:30.631305 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 02:37:30.631412 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:37:30.632762 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:30.618792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:44905)
0619 02:37:30.619253 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:30.619297 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:30.620510 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:30.621112 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:30.621119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:30.621477 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:30.622229 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:30.622246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.623311 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.623314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:30.624341 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:30.624351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.624900 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.624905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:30.625076 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:30.625714 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:30.625731 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:30.627408 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:30.629351 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:30.629359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:30.629386 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:30.629805 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:30.630073 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:30.630079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:30.630083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:30.630267 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:30.630644 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:30.630652 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:30.631004 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 02:37:30.631248 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 02:37:30.631660 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":124,"threads_started":1}
I20260619 02:37:30.634084 23582 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38615
I20260619 02:37:30.634353 23582 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:30.634883 23582 heartbeater.cc:511] Master 127.7.194.254:38615 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:37:30.636586 23391 ts_manager.cc:194] Registered new tserver with Master: e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627)
I20260619 02:37:30.638181 23391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:44905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:37:30.652632 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38615
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:30.765138 23587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:30.765455 23587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:30.765538 23587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:30.769747 23587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:30.769897 23587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:30.770025 23587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:37:30.775336 23587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:30.776481 23587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:30.777613 23587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:30.787636 23592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:30.787622 23593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:30.787755 23587 server_base.cc:1034] running on GCE node
W20260619 02:37:30.787998 23595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:30.788425 23587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:30.789120 23587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:30.790302 23587 hybrid_clock.cc:648] HybridClock initialized: now 1781836650790283 us; error 33 us; skew 500 ppm
Jun 19 02:37:30 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836650, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:30.793076 23587 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:37:30.794004 23587 webserver.cc:460] Webserver started at http://127.7.194.194:46091/ using document root <none> and password file <none>
I20260619 02:37:30.794587 23587 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:30.794675 23587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:30.794900 23587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:30.796948 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "4ae79ba41e814bff9ea30d9fdd1007ad"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "354d8b46a6df2a7bfd1295aa9bd055dd"
server_key_iv: "136bf306c95ae4a3b75b996b8f028dd6"
server_key_version: "encryptionkey@0"
I20260619 02:37:30.797537 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "4ae79ba41e814bff9ea30d9fdd1007ad"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "354d8b46a6df2a7bfd1295aa9bd055dd"
server_key_iv: "136bf306c95ae4a3b75b996b8f028dd6"
server_key_version: "encryptionkey@0"
I20260619 02:37:30.801290 23587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:37:30.803854 23602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:30.805120 23587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:37:30.805253 23587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "4ae79ba41e814bff9ea30d9fdd1007ad"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "354d8b46a6df2a7bfd1295aa9bd055dd"
server_key_iv: "136bf306c95ae4a3b75b996b8f028dd6"
server_key_version: "encryptionkey@0"
I20260619 02:37:30.805370 23587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:30.823664 23587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:30.826825 23587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:30.827023 23587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:30.827605 23587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:30.828433 23587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:30.828500 23587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:30.828569 23587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:30.828616 23587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:30.837405 23587 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:46093
I20260619 02:37:30.837423 23715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:46093 every 8 connection(s)
I20260619 02:37:30.838361 23587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:37:30.839327 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 23587
I20260619 02:37:30.839434 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:37:30.839779 7947 external_mini_cluster.cc:1428] Setting key 1f67a16c8cf50051d738bf80b1fa7ff7
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
Jun 19 02:37:30 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836650, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:30.857218 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:30.841373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:34513 (local address 127.7.194.254:38615)
0619 02:37:30.841561 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:30.841566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:30.842644 (+ 1078us) server_negotiation.cc:407] Connection header received
0619 02:37:30.843981 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:30.843984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:30.844037 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:30.844143 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:30.845335 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.845991 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.847153 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.847384 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.847498 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:30.852784 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:30.852787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:30.852789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:30.852817 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:30.854652 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:30.855320 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:30.855327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:30.855332 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:30.855436 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:30.855793 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:30.855799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:30.855804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:30.856098 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:30.856218 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:30.856791 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 02:37:30.856955 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:37:30.858016 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:30.841969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:34513)
0619 02:37:30.842493 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:30.842536 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:30.843768 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:30.844331 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:30.844338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:30.844704 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:30.845187 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:30.845200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.846121 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.846125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:30.847024 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:30.847033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:30.847538 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:30.847542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:30.847727 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:30.848356 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:30.848373 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:30.852568 (+ 4195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:30.854839 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:30.854845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:30.854860 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:30.855228 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:30.855533 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:30.855535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:30.855536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:30.855684 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:30.856229 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:30.856237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:30.856528 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 02:37:30.856699 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 02:37:30.857021 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260619 02:37:30.859413 23716 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38615
I20260619 02:37:30.859684 23716 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:30.860203 23716 heartbeater.cc:511] Master 127.7.194.254:38615 requested a full tablet report, sending...
I20260619 02:37:30.861370 23391 ts_manager.cc:194] Registered new tserver with Master: 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093)
I20260619 02:37:30.862128 23391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:34513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:37:30.868800 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38615
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:30.982131 23721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:30.982455 23721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:30.982529 23721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:30.986675 23721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:30.986763 23721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:30.986858 23721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:37:30.992229 23721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:30.993373 23721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:30.994515 23721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:31.004832 23726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:31.004838 23727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:31.004801 23729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:31.004974 23721 server_base.cc:1034] running on GCE node
I20260619 02:37:31.005363 23721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:31.006129 23721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:31.007313 23721 hybrid_clock.cc:648] HybridClock initialized: now 1781836651007296 us; error 31 us; skew 500 ppm
Jun 19 02:37:31 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836651, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:31.010278 23721 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:37:31.011168 23721 webserver.cc:460] Webserver started at http://127.7.194.195:46317/ using document root <none> and password file <none>
I20260619 02:37:31.011751 23721 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:31.011842 23721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:31.012077 23721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:37:31.014227 23721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "cacdc9fc199e49c38a88c94cd1e5158a"
format_stamp: "Formatted at 2026-06-19 02:37:31 on dist-test-slave-9tkz"
server_key: "9d977c24985fa99117ca2086e0f2474b"
server_key_iv: "0671a60c93e4f3f05f2a72e7259c960e"
server_key_version: "encryptionkey@0"
I20260619 02:37:31.014817 23721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "cacdc9fc199e49c38a88c94cd1e5158a"
format_stamp: "Formatted at 2026-06-19 02:37:31 on dist-test-slave-9tkz"
server_key: "9d977c24985fa99117ca2086e0f2474b"
server_key_iv: "0671a60c93e4f3f05f2a72e7259c960e"
server_key_version: "encryptionkey@0"
I20260619 02:37:31.018707 23721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:37:31.021176 23736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:31.022581 23721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:37:31.022711 23721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "cacdc9fc199e49c38a88c94cd1e5158a"
format_stamp: "Formatted at 2026-06-19 02:37:31 on dist-test-slave-9tkz"
server_key: "9d977c24985fa99117ca2086e0f2474b"
server_key_iv: "0671a60c93e4f3f05f2a72e7259c960e"
server_key_version: "encryptionkey@0"
I20260619 02:37:31.022826 23721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:31.041096 23721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:31.044435 23721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:31.044667 23721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:31.045305 23721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:31.046200 23721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:31.046275 23721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:31.046350 23721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:31.046398 23721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:31.055716 23721 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:40329
I20260619 02:37:31.055742 23849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:40329 every 8 connection(s)
I20260619 02:37:31.056660 23721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:37:31.065830 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 23721
I20260619 02:37:31.065992 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:37:31.066298 7947 external_mini_cluster.cc:1428] Setting key b7bd560eb27583bb3de00aaccad86d61
Jun 19 02:37:31 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836651, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:31.071414 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.058591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:40221 (local address 127.7.194.254:38615)
0619 02:37:31.058736 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.058739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.059625 (+ 886us) server_negotiation.cc:407] Connection header received
0619 02:37:31.061024 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.061028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.061091 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.061197 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:31.062560 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.063105 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.064295 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.064529 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.064640 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.068003 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:31.068006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:31.068008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:31.068033 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:31.069492 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:31.070041 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:31.070044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:31.070045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:31.070117 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:31.070437 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:31.070440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:31.070441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:31.070615 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:31.070690 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.071181 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.071280 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:37:31.072244 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.058972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:40221)
0619 02:37:31.059455 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:31.059500 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:31.060765 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:31.061387 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:31.061395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:31.061891 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:31.062409 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.062424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.063241 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.063250 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.064156 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.064165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.064686 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.064690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.064871 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.065730 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:31.065755 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:31.067836 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:31.069642 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:31.069647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:31.069660 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:31.069957 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:31.070197 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:31.070199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:31.070200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:31.070355 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:31.070698 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:31.070703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:31.070975 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 02:37:31.071135 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 02:37:31.071397 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260619 02:37:31.073323 23850 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38615
I20260619 02:37:31.073590 23850 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:31.074090 23850 heartbeater.cc:511] Master 127.7.194.254:38615 requested a full tablet report, sending...
I20260619 02:37:31.075071 23391 ts_manager.cc:194] Registered new tserver with Master: cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329)
I20260619 02:37:31.075645 23391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:40221
I20260619 02:37:31.080389 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:37:31.090131 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.082425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.7.194.254:38615)
0619 02:37:31.082588 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.082594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.082726 (+ 132us) server_negotiation.cc:407] Connection header received
0619 02:37:31.082823 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.082826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.082865 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.082933 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:31.083571 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.084066 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.085006 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.085209 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.085279 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.086856 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:31.086859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:31.086861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:31.086884 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:31.088302 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:31.088846 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:31.088848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:31.088850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:31.088926 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:31.089229 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:31.089233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:31.089235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:31.089481 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:31.089660 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.089903 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.090008 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:37:31.093374 23391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:37:31.096534 23391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:37:31.111852 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.105347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:40329 (local address 127.0.0.1:32992)
0619 02:37:31.105755 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:31.105804 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:31.105938 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:31.106819 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:31.106826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:31.106856 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:31.106993 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.107004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.108154 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.108160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.110260 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.110271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.111259 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.111263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.111365 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.111401 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:37:31.111509 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 02:37:31.111660 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":82,"thread_start_us":109,"threads_started":1}
I20260619 02:37:31.111850 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.105476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.7.194.195:40329)
0619 02:37:31.106045 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.106057 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.106082 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:37:31.106165 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.106170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.106399 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.106625 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:31.107137 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.108032 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.110412 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.111133 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.111259 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.111421 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.111558 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.111670 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":213,"threads_started":1}
I20260619 02:37:31.112195 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.106519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38627 (local address 127.0.0.1:60054)
0619 02:37:31.106849 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:31.106865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:31.106960 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:31.107666 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:31.107669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:31.107687 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:31.107787 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.107794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.109359 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.109362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.110565 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.110573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.111787 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.111791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.111896 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.111928 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 02:37:31.111986 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 02:37:31.112052 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":83,"threads_started":1}
I20260619 02:37:31.112442 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.106171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:46093 (local address 127.0.0.1:32872)
0619 02:37:31.106469 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:31.106484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:31.106568 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:31.107479 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:31.107482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:31.107500 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:31.107592 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.107597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.109059 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.109063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.110244 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.110254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.111502 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.111506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.111605 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.111626 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:37:31.112211 (+ 585us) client_negotiation.cc:240] Negotiation successful
0619 02:37:31.112295 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":25,"thread_start_us":116,"threads_started":1}
I20260619 02:37:31.112929 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.106335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.7.194.194:46093)
0619 02:37:31.106986 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.107000 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.107026 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:37:31.107109 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.107115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.107326 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.107487 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:31.107673 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.108890 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.110383 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.111160 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.111296 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.111450 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.112695 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.112795 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":124,"threads_started":1}
I20260619 02:37:31.112954 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.106902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.7.194.193:38627)
0619 02:37:31.107208 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.107224 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.107257 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:37:31.107332 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.107338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.107503 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.107949 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:31.108121 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.109209 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.110692 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.111608 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.112568 (+ 960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.112711 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.112763 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.112837 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260619 02:37:31.115681 23784 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdb6011bc06404ea1302a6f72bce5ca (DEFAULT_TABLE table=test-table [id=81ad92e40cd54246abe57f6b8078790b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:31.115682 23650 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdb6011bc06404ea1302a6f72bce5ca (DEFAULT_TABLE table=test-table [id=81ad92e40cd54246abe57f6b8078790b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:31.116856 23650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdb6011bc06404ea1302a6f72bce5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:31.116870 23784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdb6011bc06404ea1302a6f72bce5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:31.117686 23516 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdb6011bc06404ea1302a6f72bce5ca (DEFAULT_TABLE table=test-table [id=81ad92e40cd54246abe57f6b8078790b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:31.118777 23516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdb6011bc06404ea1302a6f72bce5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:31.125237 23867 tablet_bootstrap.cc:492] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: Bootstrap starting.
I20260619 02:37:31.126351 23869 tablet_bootstrap.cc:492] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Bootstrap starting.
I20260619 02:37:31.126962 23868 tablet_bootstrap.cc:492] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: Bootstrap starting.
I20260619 02:37:31.128654 23869 tablet_bootstrap.cc:654] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:31.129012 23867 tablet_bootstrap.cc:654] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:31.129109 23868 tablet_bootstrap.cc:654] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:31.129849 23869 log.cc:824] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:31.129993 23867 log.cc:824] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:31.130311 23868 log.cc:824] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:31.131661 23869 tablet_bootstrap.cc:492] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: No bootstrap required, opened a new log
I20260619 02:37:31.131873 23869 ts_tablet_manager.cc:1397] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:37:31.132187 23868 tablet_bootstrap.cc:492] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: No bootstrap required, opened a new log
I20260619 02:37:31.132398 23868 ts_tablet_manager.cc:1397] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:37:31.135121 23867 tablet_bootstrap.cc:492] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: No bootstrap required, opened a new log
I20260619 02:37:31.135334 23867 ts_tablet_manager.cc:1397] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 02:37:31.135231 23869 raft_consensus.cc:348] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.135524 23869 raft_consensus.cc:374] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:31.135597 23869 raft_consensus.cc:729] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dae38bb20144c8af479b4a960655ae, State: Initialized, Role: FOLLOWER
I20260619 02:37:31.136066 23869 consensus_queue.cc:260] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.136057 23868 raft_consensus.cc:348] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.136427 23868 raft_consensus.cc:374] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:31.136679 23582 heartbeater.cc:503] Master 127.7.194.254:38615 was elected leader, sending a full tablet report...
I20260619 02:37:31.137014 23869 ts_tablet_manager.cc:1428] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:37:31.137610 23868 raft_consensus.cc:729] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cacdc9fc199e49c38a88c94cd1e5158a, State: Initialized, Role: FOLLOWER
I20260619 02:37:31.138181 23868 consensus_queue.cc:260] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.139135 23868 ts_tablet_manager.cc:1428] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 02:37:31.139308 23867 raft_consensus.cc:348] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.139556 23867 raft_consensus.cc:374] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:31.139638 23867 raft_consensus.cc:729] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae79ba41e814bff9ea30d9fdd1007ad, State: Initialized, Role: FOLLOWER
I20260619 02:37:31.140137 23850 heartbeater.cc:503] Master 127.7.194.254:38615 was elected leader, sending a full tablet report...
I20260619 02:37:31.140120 23867 consensus_queue.cc:260] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.141119 23716 heartbeater.cc:503] Master 127.7.194.254:38615 was elected leader, sending a full tablet report...
I20260619 02:37:31.141196 23867 ts_tablet_manager.cc:1428] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 02:37:31.307698 23851 tablet.cc:2369] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:37:31.339565 23717 tablet.cc:2369] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:37:31.367950 23583 tablet.cc:2369] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:37:31.379006 23874 raft_consensus.cc:482] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:37:31.379317 23874 raft_consensus.cc:504] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.380604 23874 leader_election.cc:288] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093), e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627)
I20260619 02:37:31.380884 23875 raft_consensus.cc:482] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:37:31.381086 23875 raft_consensus.cc:504] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.382429 23875 leader_election.cc:288] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329), e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627)
I20260619 02:37:31.386085 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.381425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:56601 (local address 127.7.194.193:38627)
0619 02:37:31.381658 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.381663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.381786 (+ 123us) server_negotiation.cc:407] Connection header received
0619 02:37:31.381920 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.381923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.381980 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.382069 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:31.382358 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.383085 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.384663 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.385344 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.385450 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.385503 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.385861 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.385947 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":141}
I20260619 02:37:31.386301 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.380963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:46093 (local address 127.7.194.195:58795)
0619 02:37:31.381339 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:31.381355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:31.381499 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:31.381937 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:31.381940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:31.381953 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:31.382077 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.382083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.383073 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.383077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.384727 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.384739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.385854 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.385859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.385994 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.386020 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:37:31.386089 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:37:31.386162 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":434}
I20260619 02:37:31.386608 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.381256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38627 (local address 127.7.194.195:56601)
0619 02:37:31.381714 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:31.381726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:31.381806 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:31.382073 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:31.382076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:31.382089 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:31.382221 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.382227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.383188 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.383194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.384517 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.384525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.385593 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.385596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.385692 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.385716 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:37:31.386372 (+ 656us) client_negotiation.cc:240] Negotiation successful
0619 02:37:31.386452 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":3,"thread_start_us":155,"threads_started":1}
I20260619 02:37:31.386965 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.381030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:58795 (local address 127.7.194.194:46093)
0619 02:37:31.381219 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.381225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.381596 (+ 371us) server_negotiation.cc:407] Connection header received
0619 02:37:31.381719 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.381725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.381854 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.381932 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:31.382202 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.382948 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.384916 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.385628 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.385732 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.385779 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.386771 (+ 992us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.386852 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":84}
I20260619 02:37:31.387125 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.382716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:56831 (local address 127.7.194.195:40329)
0619 02:37:31.383118 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.383122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.383132 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:37:31.383163 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.383165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.383201 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.383263 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:31.383581 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.384290 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.385838 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.386407 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.386510 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.386551 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.386950 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.387011 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":340}
I20260619 02:37:31.387147 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.382569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:40329 (local address 127.7.194.194:56831)
0619 02:37:31.382867 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:31.382879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:31.382964 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:31.383353 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:31.383356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:31.383376 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:31.383482 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.383489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.384414 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.384417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.385707 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.385717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.386694 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.386699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.386809 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.386833 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:37:31.386935 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:37:31.387005 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":48,"thread_start_us":74,"threads_started":1}
I20260619 02:37:31.387564 23670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdb6011bc06404ea1302a6f72bce5ca" candidate_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" is_pre_election: true
I20260619 02:37:31.387732 23804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdb6011bc06404ea1302a6f72bce5ca" candidate_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" is_pre_election: true
I20260619 02:37:31.387809 23670 raft_consensus.cc:2454] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cacdc9fc199e49c38a88c94cd1e5158a in term 0.
I20260619 02:37:31.387916 23804 raft_consensus.cc:2454] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae79ba41e814bff9ea30d9fdd1007ad in term 0.
I20260619 02:37:31.388432 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.382902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38627 (local address 127.7.194.194:44801)
0619 02:37:31.383352 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:31.383365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:31.383452 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:31.384578 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:31.384581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:31.384597 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:31.384786 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.384793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.385875 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.385877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.386990 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:31.387001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.388017 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.388021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:31.388120 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.388148 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:37:31.388224 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:37:31.388292 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":56,"thread_start_us":59,"threads_started":1}
I20260619 02:37:31.388394 23603 leader_election.cc:302] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae79ba41e814bff9ea30d9fdd1007ad, cacdc9fc199e49c38a88c94cd1e5158a; no voters:
I20260619 02:37:31.388515 23879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.383573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:44801 (local address 127.7.194.193:38627)
0619 02:37:31.384009 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.384018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.384033 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:37:31.384074 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.384077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.384133 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.384480 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:31.384929 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.385760 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.387132 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.387873 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.388011 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.388057 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.388254 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.388328 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":364,"thread_start_us":165,"threads_started":1}
I20260619 02:37:31.388463 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdb6011bc06404ea1302a6f72bce5ca" candidate_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dae38bb20144c8af479b4a960655ae" is_pre_election: true
I20260619 02:37:31.388691 23875 raft_consensus.cc:2781] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:37:31.388747 23536 raft_consensus.cc:2454] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cacdc9fc199e49c38a88c94cd1e5158a in term 0.
I20260619 02:37:31.388788 23875 raft_consensus.cc:482] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:37:31.388833 23875 raft_consensus.cc:3037] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:31.389020 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdb6011bc06404ea1302a6f72bce5ca" candidate_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dae38bb20144c8af479b4a960655ae" is_pre_election: true
I20260619 02:37:31.389166 23535 raft_consensus.cc:2454] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae79ba41e814bff9ea30d9fdd1007ad in term 0.
I20260619 02:37:31.389369 23737 leader_election.cc:302] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cacdc9fc199e49c38a88c94cd1e5158a, e7dae38bb20144c8af479b4a960655ae; no voters:
I20260619 02:37:31.389647 23874 raft_consensus.cc:2781] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:37:31.389740 23874 raft_consensus.cc:482] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:37:31.389791 23874 raft_consensus.cc:3037] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:31.390453 23875 raft_consensus.cc:504] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.390904 23875 leader_election.cc:288] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [CANDIDATE]: Term 1 election: Requested vote from peers cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329), e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627)
I20260619 02:37:31.391227 23874 raft_consensus.cc:504] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.391311 23804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdb6011bc06404ea1302a6f72bce5ca" candidate_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cacdc9fc199e49c38a88c94cd1e5158a"
I20260619 02:37:31.391345 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdb6011bc06404ea1302a6f72bce5ca" candidate_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dae38bb20144c8af479b4a960655ae"
I20260619 02:37:31.391441 23535 raft_consensus.cc:3037] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:31.391510 23804 raft_consensus.cc:2379] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ae79ba41e814bff9ea30d9fdd1007ad in current term 1: Already voted for candidate cacdc9fc199e49c38a88c94cd1e5158a in this term.
I20260619 02:37:31.391572 23874 leader_election.cc:288] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [CANDIDATE]: Term 1 election: Requested vote from peers 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093), e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627)
I20260619 02:37:31.392601 23535 raft_consensus.cc:2454] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae79ba41e814bff9ea30d9fdd1007ad in term 1.
I20260619 02:37:31.392931 23603 leader_election.cc:302] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ae79ba41e814bff9ea30d9fdd1007ad, e7dae38bb20144c8af479b4a960655ae; no voters: cacdc9fc199e49c38a88c94cd1e5158a
I20260619 02:37:31.393199 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdb6011bc06404ea1302a6f72bce5ca" candidate_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dae38bb20144c8af479b4a960655ae"
I20260619 02:37:31.393385 23535 raft_consensus.cc:2379] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cacdc9fc199e49c38a88c94cd1e5158a in current term 1: Already voted for candidate 4ae79ba41e814bff9ea30d9fdd1007ad in this term.
I20260619 02:37:31.393400 23670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdb6011bc06404ea1302a6f72bce5ca" candidate_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad"
I20260619 02:37:31.393628 23670 raft_consensus.cc:2379] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cacdc9fc199e49c38a88c94cd1e5158a in current term 1: Already voted for candidate 4ae79ba41e814bff9ea30d9fdd1007ad in this term.
I20260619 02:37:31.393764 23875 raft_consensus.cc:2781] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:37:31.393990 23737 leader_election.cc:302] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cacdc9fc199e49c38a88c94cd1e5158a; no voters: 4ae79ba41e814bff9ea30d9fdd1007ad, e7dae38bb20144c8af479b4a960655ae
I20260619 02:37:31.394003 23875 raft_consensus.cc:686] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 LEADER]: Becoming Leader. State: Replica: 4ae79ba41e814bff9ea30d9fdd1007ad, State: Running, Role: LEADER
I20260619 02:37:31.394167 23874 raft_consensus.cc:2726] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 02:37:31.394392 23875 consensus_queue.cc:237] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:31.397322 23391 catalog_manager.cc:5495] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194). New cstate: current_term: 1 leader_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } health_report { overall_health: UNKNOWN } } }
I20260619 02:37:31.411680 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.407997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.7.194.194:46093)
0619 02:37:31.408140 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:31.408145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:31.408159 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:37:31.408322 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:31.408325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:31.408388 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:31.408500 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:37:31.408753 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.409294 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.410201 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:31.410423 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:31.410528 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:31.410845 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:37:31.411269 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:37:31.411432 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:31.411469 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 02:37:31.411526 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:37:31.417496 23804 raft_consensus.cc:1261] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 1 FOLLOWER]: Refusing update from remote peer 4ae79ba41e814bff9ea30d9fdd1007ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:31.417510 23535 raft_consensus.cc:1261] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Refusing update from remote peer 4ae79ba41e814bff9ea30d9fdd1007ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:31.418146 23880 consensus_queue.cc:1035] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:31.418318 23875 consensus_queue.cc:1035] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:31.427541 23884 mvcc.cc:205] Tried to move back new op lower bound from 7298402924199104512 to 7298402924114325504. Current Snapshot: MvccSnapshot[applied={T|T < 7298402924199104512}]
I20260619 02:37:31.430102 23885 mvcc.cc:205] Tried to move back new op lower bound from 7298402924199104512 to 7298402924114325504. Current Snapshot: MvccSnapshot[applied={T|T < 7298402924199104512 or (T in {7298402924199104512})}]
I20260619 02:37:31.436239 23391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:31.436441 23391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:31.439132 23391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Sending DeleteTablet for 3 replicas of tablet 4cdb6011bc06404ea1302a6f72bce5ca
I20260619 02:37:31.440086 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 23453
I20260619 02:37:31.440069 23650 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdb6011bc06404ea1302a6f72bce5ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:31 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:32872
I20260619 02:37:31.440243 23784 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdb6011bc06404ea1302a6f72bce5ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:31 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:32992
I20260619 02:37:31.440817 23894 tablet_replica.cc:330] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: stopping tablet replica
I20260619 02:37:31.441184 23894 raft_consensus.cc:2229] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:31.441537 23894 raft_consensus.cc:2258] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:31.441896 23893 tablet_replica.cc:330] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: stopping tablet replica
I20260619 02:37:31.442162 23893 raft_consensus.cc:2229] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:37:31.442490 23893 raft_consensus.cc:2258] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:31.445636 23894 ts_tablet_manager.cc:1905] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:37:31.445636 23893 ts_tablet_manager.cc:1905] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:37:31.448578 23893 ts_tablet_manager.cc:1918] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:31.448786 23893 log.cc:1196] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/4cdb6011bc06404ea1302a6f72bce5ca
I20260619 02:37:31.448858 23894 ts_tablet_manager.cc:1918] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:31.448942 23894 log.cc:1196] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/4cdb6011bc06404ea1302a6f72bce5ca
I20260619 02:37:31.449287 23894 ts_tablet_manager.cc:1939] T 4cdb6011bc06404ea1302a6f72bce5ca P cacdc9fc199e49c38a88c94cd1e5158a: Deleting consensus metadata
I20260619 02:37:31.449287 23893 ts_tablet_manager.cc:1939] T 4cdb6011bc06404ea1302a6f72bce5ca P 4ae79ba41e814bff9ea30d9fdd1007ad: Deleting consensus metadata
W20260619 02:37:31.450656 23375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:38627 (error 108)
I20260619 02:37:31.450896 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 23587
I20260619 02:37:31.451176 23375 catalog_manager.cc:4841] TS 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093): tablet 4cdb6011bc06404ea1302a6f72bce5ca (table test-table [id=81ad92e40cd54246abe57f6b8078790b]) successfully deleted
I20260619 02:37:31.451318 23375 catalog_manager.cc:4841] TS cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329): tablet 4cdb6011bc06404ea1302a6f72bce5ca (table test-table [id=81ad92e40cd54246abe57f6b8078790b]) successfully deleted
I20260619 02:37:31.452344 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:31.452070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38627 (local address 127.0.0.1:60062)
0619 02:37:31.452196 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:31.452265 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:37:31.452487 23375 catalog_manager.cc:4584] TS e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cdb6011bc06404ea1302a6f72bce5ca: Network error: Client connection negotiation failed: client connection to 127.7.194.193:38627: connect: Connection refused (error 111)
I20260619 02:37:31.457257 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 23721
I20260619 02:37:31.463013 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 23359
I20260619 02:37:51.470752 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38615
--webserver_interface=127.7.194.254
--webserver_port=32927
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:37:51.583959 23896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:51.584219 23896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:51.584295 23896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:51.588289 23896 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:37:51.588340 23896 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:37:51.588356 23896 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:37:51.588369 23896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:37:51.588387 23896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:51.588403 23896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:51.588418 23896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:37:51.588433 23896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:37:51.593526 23896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:51.594647 23896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:51.595680 23896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:51.608171 23903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:51.608286 23896 server_base.cc:1034] running on GCE node
W20260619 02:37:51.608198 23902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:51.608171 23905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:51.609045 23896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:51.610253 23896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:51.611413 23896 hybrid_clock.cc:648] HybridClock initialized: now 1781836671611378 us; error 50 us; skew 500 ppm
Jun 19 02:37:51 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836671, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:51.618945 23896 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:37:51.619918 23896 webserver.cc:460] Webserver started at http://127.7.194.254:32927/ using document root <none> and password file <none>
I20260619 02:37:51.620530 23896 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:51.620644 23896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:51.624625 23896 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:37:51.626819 23912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:51.628033 23896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:37:51.628183 23896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "7154e3ae52f34384900e9d44943ea048"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "eb9d5850581a0bf12c41a38c5a536f87"
server_key_iv: "1a7c069e6848430b6923e0e7b2b89944"
server_key_version: "encryptionkey@0"
I20260619 02:37:51.628777 23896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:51.649219 23896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:51.652127 23896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:51.652346 23896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:51.659786 23896 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:38615
I20260619 02:37:51.660115 23964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:38615 every 8 connection(s)
I20260619 02:37:51.661123 23896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:37:51.665516 23965 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:37:51.667153 23965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Bootstrap starting.
I20260619 02:37:51.667893 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 23896
I20260619 02:37:51.668330 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:38627
--local_ip_for_outbound_sockets=127.7.194.193
--tserver_master_addrs=127.7.194.254:38615
--webserver_port=43683
--webserver_interface=127.7.194.193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:51.684413 23965 log.cc:824] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:51.693118 23965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:37:51.693645 23965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Bootstrap complete.
I20260619 02:37:51.697798 23965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:37:51.698208 23965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7154e3ae52f34384900e9d44943ea048, State: Initialized, Role: FOLLOWER
I20260619 02:37:51.698761 23965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:37:51.698904 23965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:37:51.698975 23965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:37:51.699083 23965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:37:51.700991 23965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:37:51.701351 23965 leader_election.cc:302] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7154e3ae52f34384900e9d44943ea048; no voters:
I20260619 02:37:51.701726 23965 leader_election.cc:288] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:37:51.701844 23969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:37:51.702065 23969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 2 LEADER]: Becoming Leader. State: Replica: 7154e3ae52f34384900e9d44943ea048, State: Running, Role: LEADER
I20260619 02:37:51.702425 23969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:37:51.702996 23965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:37:51.703910 23970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7154e3ae52f34384900e9d44943ea048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } } }
I20260619 02:37:51.704030 23970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:51.704361 23978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:37:51.704411 23971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7154e3ae52f34384900e9d44943ea048. Latest consensus state: current_term: 2 leader_uuid: "7154e3ae52f34384900e9d44943ea048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } } }
I20260619 02:37:51.704504 23971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: This master's current role is: LEADER
I20260619 02:37:51.709334 23978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:37:51.710124 23978 catalog_manager.cc:1259] Loaded cluster ID: 88a46d67c68e4c288db67f48a703f80b
I20260619 02:37:51.710208 23978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:37:51.711959 23978 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:37:51.712613 23978 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Loaded TSK: 0
I20260619 02:37:51.725525 23978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Generated new TSK 1
I20260619 02:37:51.725878 23978 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:37:51.791980 23967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:51.792222 23967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:51.792260 23967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:51.796277 23967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:51.796352 23967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:51.796408 23967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:37:51.801524 23967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:38627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=43683
--tserver_master_addrs=127.7.194.254:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:51.802584 23967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:51.803639 23967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:51.813591 23995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:51.813580 23993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:51.813642 23992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:51.813954 23967 server_base.cc:1034] running on GCE node
I20260619 02:37:51.814498 23967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:51.815268 23967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:51.816463 23967 hybrid_clock.cc:648] HybridClock initialized: now 1781836671816442 us; error 33 us; skew 500 ppm
Jun 19 02:37:51 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836671, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:51.819764 23967 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:37:51.820724 23967 webserver.cc:460] Webserver started at http://127.7.194.193:43683/ using document root <none> and password file <none>
I20260619 02:37:51.821310 23967 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:51.821386 23967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:51.825400 23967 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:37:51.827804 24002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:51.829142 23967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:37:51.829288 23967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "e7dae38bb20144c8af479b4a960655ae"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "f60eb4e8ccc1ba4a0ab74b811c6bb4b4"
server_key_iv: "77a7d7f2108cb6bd2678ce4d5b7269a1"
server_key_version: "encryptionkey@0"
I20260619 02:37:51.829907 23967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:51.849335 23967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:51.852854 23967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:51.853077 23967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:51.853799 23967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:51.854975 24009 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:37:51.859387 23967 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:37:51.859476 23967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 02:37:51.859557 23967 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:37:51.861248 23967 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:37:51.861339 23967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 02:37:51.861697 24009 tablet_bootstrap.cc:492] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Bootstrap starting.
I20260619 02:37:51.871917 23967 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:38627
I20260619 02:37:51.872021 24116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:38627 every 8 connection(s)
I20260619 02:37:51.872900 23967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:37:51.875095 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 23967
I20260619 02:37:51.875617 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:46093
--local_ip_for_outbound_sockets=127.7.194.194
--tserver_master_addrs=127.7.194.254:38615
--webserver_port=46091
--webserver_interface=127.7.194.194
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:51.882040 24009 log.cc:824] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:37:51 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836671, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:51.891530 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:51.875018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:53901 (local address 127.7.194.254:38615)
0619 02:37:51.875456 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:51.875476 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:51.877381 (+ 1905us) server_negotiation.cc:407] Connection header received
0619 02:37:51.878570 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:51.878581 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:51.878884 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:51.879213 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:51.880582 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:51.881871 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:51.883327 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:51.883815 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:51.883995 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:51.886514 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:51.886519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:51.886531 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:51.886559 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:51.888258 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:51.888799 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:51.888804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:51.888807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:51.888921 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:51.889279 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:51.889283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:51.889287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:51.889696 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:51.889843 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:51.890400 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 02:37:51.890654 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":126,"threads_started":1}
I20260619 02:37:51.891615 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:51.874924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:53901)
0619 02:37:51.877187 (+ 2263us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:51.877241 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:51.878138 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:51.879274 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:51.879282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:51.879868 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:51.880400 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:51.880415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:51.882047 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:51.882050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:51.883197 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:51.883206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:51.883964 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:51.883968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:51.884155 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:51.884789 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:51.884806 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:51.886316 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:51.888388 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:51.888394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:51.888406 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:51.888662 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:51.889019 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:51.889024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:51.889026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:51.889164 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:51.889815 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:51.889821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:51.890078 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 02:37:51.890284 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 02:37:51.890620 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2067,"thread_start_us":132,"threads_started":1}
I20260619 02:37:51.892041 24009 tablet_bootstrap.cc:492] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:37:51.892490 24009 tablet_bootstrap.cc:492] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Bootstrap complete.
I20260619 02:37:51.894014 24009 ts_tablet_manager.cc:1397] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20260619 02:37:51.894622 24117 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38615
I20260619 02:37:51.894869 24117 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:51.895360 24117 heartbeater.cc:511] Master 127.7.194.254:38615 requested a full tablet report, sending...
I20260619 02:37:51.897058 24009 raft_consensus.cc:348] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:51.897768 24009 raft_consensus.cc:729] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dae38bb20144c8af479b4a960655ae, State: Initialized, Role: FOLLOWER
I20260619 02:37:51.897801 23929 ts_manager.cc:194] Registered new tserver with Master: e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627)
I20260619 02:37:51.898267 24009 consensus_queue.cc:260] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:51.899317 24009 ts_tablet_manager.cc:1428] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:37:51.901484 23929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:53901
I20260619 02:37:51.904156 24117 heartbeater.cc:503] Master 127.7.194.254:38615 was elected leader, sending a full tablet report...
Jun 19 02:37:51 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836671, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.193@KRBTEST.COM
I20260619 02:37:51.912251 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:51.900751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38627 (local address 127.0.0.1:55280)
0619 02:37:51.901182 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:51.901237 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:51.901826 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:51.902510 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:51.902516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:51.902983 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:51.903146 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:51.903155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:51.904432 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:51.904436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:51.905329 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:51.905343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:51.906121 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:51.906123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:51.906233 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:51.906843 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:51.906854 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:51.908194 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:51.910478 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:51.910486 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:51.910490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:51.910709 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:51.911058 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:51.911060 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:51.911061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:51.911136 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:51.911702 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:51.911708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:51.911826 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 02:37:51.911973 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 02:37:51.912120 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260619 02:37:51.912452 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:51.901532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.7.194.193:38627)
0619 02:37:51.901873 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:51.901885 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:51.901909 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:37:51.901981 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:51.901985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:51.902178 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:51.902328 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:51.903305 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:51.904296 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:51.905453 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:51.906000 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:51.906125 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:51.908341 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:51.908352 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:51.908365 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:51.908392 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:51.910365 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:51.910833 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:51.910841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:51.910845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:51.910953 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:51.911224 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:51.911231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:51.911234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:51.911574 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:51.911715 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:51.912084 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 02:37:51.912265 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":117,"threads_started":1}
I20260619 02:37:51.913427 24051 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdb6011bc06404ea1302a6f72bce5ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:31 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:55280
I20260619 02:37:51.914352 24126 tablet_replica.cc:330] stopping tablet replica
I20260619 02:37:51.914654 24126 raft_consensus.cc:2229] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:51.914911 24126 raft_consensus.cc:2258] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:51.915869 24126 ts_tablet_manager.cc:1905] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:37:51.919582 24126 ts_tablet_manager.cc:1918] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:51.919667 24126 log.cc:1196] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal/wals/4cdb6011bc06404ea1302a6f72bce5ca
I20260619 02:37:51.919965 24126 ts_tablet_manager.cc:1939] T 4cdb6011bc06404ea1302a6f72bce5ca P e7dae38bb20144c8af479b4a960655ae: Deleting consensus metadata
I20260619 02:37:51.921679 23913 catalog_manager.cc:4841] TS e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627): tablet 4cdb6011bc06404ea1302a6f72bce5ca (table test-table [id=81ad92e40cd54246abe57f6b8078790b]) successfully deleted
W20260619 02:37:52.000465 24121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:52.000789 24121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:52.000866 24121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:52.005206 24121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:52.005288 24121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:52.005378 24121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:37:52.010653 24121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:46093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=46091
--tserver_master_addrs=127.7.194.254:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:52.011724 24121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:52.012768 24121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:37:52.022195 24134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:52.022223 24132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:52.022202 24131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:52.022508 24121 server_base.cc:1034] running on GCE node
I20260619 02:37:52.022931 24121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:52.023607 24121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:52.024765 24121 hybrid_clock.cc:648] HybridClock initialized: now 1781836672024733 us; error 44 us; skew 500 ppm
Jun 19 02:37:52 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836672, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:52.027518 24121 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:37:52.028318 24121 webserver.cc:460] Webserver started at http://127.7.194.194:46091/ using document root <none> and password file <none>
I20260619 02:37:52.028841 24121 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:52.028910 24121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:52.032874 24121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:37:52.035086 24141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:52.036351 24121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:37:52.036490 24121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "4ae79ba41e814bff9ea30d9fdd1007ad"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "354d8b46a6df2a7bfd1295aa9bd055dd"
server_key_iv: "136bf306c95ae4a3b75b996b8f028dd6"
server_key_version: "encryptionkey@0"
I20260619 02:37:52.037041 24121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:52.050091 24121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:52.052857 24121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:52.053071 24121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:52.053702 24121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:52.054513 24121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:52.054585 24121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:52.054659 24121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:52.054702 24121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:52.063223 24121 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:46093
I20260619 02:37:52.063303 24254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:46093 every 8 connection(s)
I20260619 02:37:52.064248 24121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:37:52.074321 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 24121
Jun 19 02:37:52 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836672, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:52.074826 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:40329
--local_ip_for_outbound_sockets=127.7.194.195
--tserver_master_addrs=127.7.194.254:38615
--webserver_port=46317
--webserver_interface=127.7.194.195
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:37:52.079097 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.066371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:53107 (local address 127.7.194.254:38615)
0619 02:37:52.066537 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.066542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.067609 (+ 1067us) server_negotiation.cc:407] Connection header received
0619 02:37:52.068961 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.068964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.069028 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.069120 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:52.070302 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.070843 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.072016 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.072279 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.072404 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.074967 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:52.074971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:52.074974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:52.075003 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:52.076636 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:52.077275 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:52.077278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:52.077280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:52.077361 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:52.077764 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:52.077767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:52.077768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:52.077954 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:52.078061 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:52.078687 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 02:37:52.078942 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:37:52.080390 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.066839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:53107)
0619 02:37:52.067460 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:52.067501 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:52.068747 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:52.069294 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:52.069301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:52.069680 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:52.070150 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.070163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.070969 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.070973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.071886 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.071894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.072441 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.072445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.072627 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.073253 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:52.073268 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:52.074792 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:52.076758 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:52.076767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:52.076785 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:52.077162 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:52.077441 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:52.077447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:52.077452 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:52.077663 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:52.078087 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:52.078097 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:52.078426 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 02:37:52.078638 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 02:37:52.079011 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":147,"threads_started":1}
I20260619 02:37:52.081686 24255 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38615
I20260619 02:37:52.081930 24255 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:52.082430 24255 heartbeater.cc:511] Master 127.7.194.254:38615 requested a full tablet report, sending...
I20260619 02:37:52.083493 23929 ts_manager.cc:194] Registered new tserver with Master: 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093)
I20260619 02:37:52.084087 23929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:53107
W20260619 02:37:52.186892 24258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:37:52.187180 24258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:37:52.187223 24258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:37:52.191334 24258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:37:52.191395 24258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:37:52.191488 24258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:37:52.196815 24258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:40329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=46317
--tserver_master_addrs=127.7.194.254:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:37:52.198006 24258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:37:52.199061 24258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:37:52.209712 24258 server_base.cc:1034] running on GCE node
W20260619 02:37:52.209676 24264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:52.209676 24266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:37:52.210016 24263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:37:52.210469 24258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:37:52.211148 24258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:37:52.212325 24258 hybrid_clock.cc:648] HybridClock initialized: now 1781836672212307 us; error 29 us; skew 500 ppm
Jun 19 02:37:52 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836672, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:37:52.215188 24258 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:37:52.216033 24258 webserver.cc:460] Webserver started at http://127.7.194.195:46317/ using document root <none> and password file <none>
I20260619 02:37:52.216547 24258 fs_manager.cc:359] Metadata directory not provided
I20260619 02:37:52.216615 24258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:37:52.220571 24258 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 02:37:52.222642 24273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:52.223851 24258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:37:52.223941 24258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "cacdc9fc199e49c38a88c94cd1e5158a"
format_stamp: "Formatted at 2026-06-19 02:37:31 on dist-test-slave-9tkz"
server_key: "9d977c24985fa99117ca2086e0f2474b"
server_key_iv: "0671a60c93e4f3f05f2a72e7259c960e"
server_key_version: "encryptionkey@0"
I20260619 02:37:52.224439 24258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:37:52.243494 24258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:37:52.246877 24258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:37:52.247030 24258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:37:52.247560 24258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:37:52.248311 24258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:37:52.248358 24258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:52.248396 24258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:37:52.248415 24258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:37:52.257392 24258 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:40329
I20260619 02:37:52.257423 24386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:40329 every 8 connection(s)
I20260619 02:37:52.258425 24258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:37:52.262041 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 24258
Jun 19 02:37:52 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836672, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:37:52.272217 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.262764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.7.194.254:38615)
0619 02:37:52.262998 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.263003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.263385 (+ 382us) server_negotiation.cc:407] Connection header received
0619 02:37:52.263509 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.263511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.263544 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.263642 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:52.264498 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.265299 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.266498 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.266751 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.266831 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.268447 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:52.268451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:52.268454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:52.268479 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:52.270421 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:52.270918 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:52.270922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:52.270924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:52.271007 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:52.271326 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:52.271329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:52.271330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:52.271500 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:52.271683 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:52.271878 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:37:52.272001 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
I20260619 02:37:52.273756 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.260366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:48417 (local address 127.7.194.254:38615)
0619 02:37:52.260573 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.260577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.261420 (+ 843us) server_negotiation.cc:407] Connection header received
0619 02:37:52.263049 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.263056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.263112 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.263188 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:52.264778 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.265451 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.266953 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.267154 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.267262 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.269912 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:52.269915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:52.269917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:52.269950 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:52.271566 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:52.272184 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:52.272187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:52.272189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:52.272277 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:52.272679 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:52.272682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:52.272684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:52.272850 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:52.272953 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:52.273460 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 02:37:52.273605 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 02:37:52.274667 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.260765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:48417)
0619 02:37:52.261243 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:52.261299 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:52.262802 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:52.263414 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:52.263424 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:52.263991 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:37:52.264649 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.264664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.265584 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.265587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.266819 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.266835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.267307 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.267312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.267482 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.268054 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:37:52.268070 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:37:52.269764 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:37:52.271689 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:52.271695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:52.271711 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:52.272068 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:52.272402 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:37:52.272405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:37:52.272407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:37:52.272575 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:37:52.272955 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:37:52.272965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:37:52.273215 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 02:37:52.273406 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 02:37:52.273773 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
I20260619 02:37:52.275810 24387 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38615
I20260619 02:37:52.276049 24387 heartbeater.cc:464] Registering TS with master...
I20260619 02:37:52.276537 24387 heartbeater.cc:511] Master 127.7.194.254:38615 requested a full tablet report, sending...
I20260619 02:37:52.277593 23929 ts_manager.cc:194] Registered new tserver with Master: cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329)
I20260619 02:37:52.278157 23929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:48417
I20260619 02:37:52.286203 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:37:52.290653 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.286876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.7.194.254:38615)
0619 02:37:52.287050 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.287056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.287249 (+ 193us) server_negotiation.cc:407] Connection header received
0619 02:37:52.287472 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.287477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.287660 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.287834 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:37:52.288113 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.288640 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.289497 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.289749 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.289827 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.290121 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:37:52.290276 (+ 155us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 02:37:52.290496 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55116: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 02:37:52.290771 24119 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55116: token expired
W20260619 02:37:52.291080 23853 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.194.254:38615)
I20260619 02:37:52.298521 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.291453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.7.194.254:38615)
0619 02:37:52.291572 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.291578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.291609 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:37:52.291711 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.291714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.291768 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.291850 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:37:52.292452 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.292940 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.293797 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.294010 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.294117 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.295499 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:37:52.295505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:37:52.295513 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:37:52.295537 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:37:52.296821 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:52.297267 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:52.297271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:52.297273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:52.297356 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:37:52.297656 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:37:52.297660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:37:52.297662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:37:52.297846 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:37:52.297988 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:52.298230 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 02:37:52.298353 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:37:52.300994 23929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:37:52.304534 23929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:37:52.312896 24051 tablet_service.cc:1467] Processing CreateTablet for tablet 5b220efdee514ab2bfee9271266e585c (DEFAULT_TABLE table=test-table [id=087e1997bbb04c7cbf95b878ce3f9c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:52.313621 24051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b220efdee514ab2bfee9271266e585c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:52.316581 24009 tablet_bootstrap.cc:492] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Bootstrap starting.
I20260619 02:37:52.317521 24009 tablet_bootstrap.cc:654] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:52.317597 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.311922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:46093 (local address 127.0.0.1:44276)
0619 02:37:52.312209 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:52.312224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:52.312324 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:52.313039 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:52.313043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:52.313062 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:52.313187 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.313194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.314730 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.314734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.315770 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.315780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.317046 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.317050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.317143 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.317205 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 02:37:52.317286 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:37:52.317361 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":102,"thread_start_us":108,"threads_started":1}
I20260619 02:37:52.317770 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.311618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:40329 (local address 127.0.0.1:47936)
0619 02:37:52.311726 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:52.311749 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:52.311832 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:52.312980 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:52.312983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:52.313001 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:52.313162 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.313169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.314789 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.314792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.315938 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.315949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.317321 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.317325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.317439 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.317465 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:37:52.317516 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:37:52.317623 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":29}
I20260619 02:37:52.318084 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.312145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.7.194.194:46093)
0619 02:37:52.312508 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.312522 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.312545 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:37:52.312627 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.312633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.312873 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.313342 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:52.313517 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.314604 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.315898 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.316922 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.317035 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.317185 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:52.317487 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 02:37:52.317951 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":151,"threads_started":1}
I20260619 02:37:52.318413 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.311976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.7.194.195:40329)
0619 02:37:52.312571 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.312580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.312599 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:37:52.312661 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.312665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.312833 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.312998 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:52.313790 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.314654 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.316075 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.317170 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.317938 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.318084 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:52.318139 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 02:37:52.318249 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":143,"threads_started":1}
I20260619 02:37:52.319177 24009 tablet_bootstrap.cc:492] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: No bootstrap required, opened a new log
I20260619 02:37:52.319265 24009 ts_tablet_manager.cc:1397] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 02:37:52.319722 24009 raft_consensus.cc:348] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:52.319850 24009 raft_consensus.cc:374] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:52.319888 24009 raft_consensus.cc:729] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dae38bb20144c8af479b4a960655ae, State: Initialized, Role: FOLLOWER
I20260619 02:37:52.320016 24009 consensus_queue.cc:260] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:52.320289 24009 ts_tablet_manager.cc:1428] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:37:52.321444 24321 tablet_service.cc:1467] Processing CreateTablet for tablet 5b220efdee514ab2bfee9271266e585c (DEFAULT_TABLE table=test-table [id=087e1997bbb04c7cbf95b878ce3f9c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:52.321447 24189 tablet_service.cc:1467] Processing CreateTablet for tablet 5b220efdee514ab2bfee9271266e585c (DEFAULT_TABLE table=test-table [id=087e1997bbb04c7cbf95b878ce3f9c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:37:52.322469 24321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b220efdee514ab2bfee9271266e585c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:52.322515 24189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b220efdee514ab2bfee9271266e585c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:37:52.329435 24401 tablet_bootstrap.cc:492] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: Bootstrap starting.
I20260619 02:37:52.329964 24400 tablet_bootstrap.cc:492] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: Bootstrap starting.
I20260619 02:37:52.332077 24401 tablet_bootstrap.cc:654] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:52.332104 24400 tablet_bootstrap.cc:654] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: Neither blocks nor log segments found. Creating new log.
I20260619 02:37:52.332859 24400 log.cc:824] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:52.332841 24401 log.cc:824] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:37:52.334666 24400 tablet_bootstrap.cc:492] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: No bootstrap required, opened a new log
I20260619 02:37:52.334693 24401 tablet_bootstrap.cc:492] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: No bootstrap required, opened a new log
I20260619 02:37:52.334889 24400 ts_tablet_manager.cc:1397] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:37:52.334933 24401 ts_tablet_manager.cc:1397] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:37:52.337905 24401 raft_consensus.cc:348] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:52.337905 24400 raft_consensus.cc:348] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:52.338135 24401 raft_consensus.cc:374] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:52.338160 24400 raft_consensus.cc:374] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:37:52.338200 24401 raft_consensus.cc:729] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cacdc9fc199e49c38a88c94cd1e5158a, State: Initialized, Role: FOLLOWER
I20260619 02:37:52.338267 24400 raft_consensus.cc:729] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae79ba41e814bff9ea30d9fdd1007ad, State: Initialized, Role: FOLLOWER
I20260619 02:37:52.338615 24401 consensus_queue.cc:260] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:52.338690 24400 consensus_queue.cc:260] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:52.339310 24387 heartbeater.cc:503] Master 127.7.194.254:38615 was elected leader, sending a full tablet report...
I20260619 02:37:52.339337 24255 heartbeater.cc:503] Master 127.7.194.254:38615 was elected leader, sending a full tablet report...
I20260619 02:37:52.339681 24401 ts_tablet_manager.cc:1428] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:37:52.340720 24400 ts_tablet_manager.cc:1428] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:37:52.349829 24404 raft_consensus.cc:482] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:37:52.350039 24404 raft_consensus.cc:504] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:52.351383 24404 leader_election.cc:288] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093), e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627)
I20260619 02:37:52.355316 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.351734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:38365 (local address 127.7.194.194:46093)
0619 02:37:52.351887 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.351892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.351907 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:37:52.351964 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.351969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.352024 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.352103 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:52.352381 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.353061 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.354268 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.354829 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.354917 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.354957 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:52.355164 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 02:37:52.355219 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:37:52.355362 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.351590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:46093 (local address 127.7.194.195:38365)
0619 02:37:52.351716 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:52.351739 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:52.351832 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:52.352128 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:52.352134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:52.352152 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:52.352268 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.352273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.353159 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.353162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.354146 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.354154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.354931 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.354934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.355042 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.355066 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:37:52.355145 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:37:52.355214 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:37:52.356000 24209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b220efdee514ab2bfee9271266e585c" candidate_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" is_pre_election: true
I20260619 02:37:52.356139 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.351969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:57321 (local address 127.7.194.193:38627)
0619 02:37:52.352110 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.352114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.352387 (+ 273us) server_negotiation.cc:407] Connection header received
0619 02:37:52.352478 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.352480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.352531 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.352605 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:37:52.352873 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.353547 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.354718 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.355313 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.355453 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.355627 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:52.355968 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 02:37:52.356033 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:37:52.356148 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.351866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38627 (local address 127.7.194.195:57321)
0619 02:37:52.352278 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:52.352289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:37:52.352399 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:37:52.352638 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:37:52.352641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:37:52.352657 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:37:52.352770 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.352776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.353670 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.353673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.354597 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:37:52.354607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.355745 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.355749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:37:52.355842 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.355863 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:37:52.355939 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:37:52.356005 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260619 02:37:52.356348 24209 raft_consensus.cc:2454] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cacdc9fc199e49c38a88c94cd1e5158a in term 0.
I20260619 02:37:52.356645 24070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b220efdee514ab2bfee9271266e585c" candidate_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dae38bb20144c8af479b4a960655ae" is_pre_election: true
I20260619 02:37:52.356818 24274 leader_election.cc:302] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae79ba41e814bff9ea30d9fdd1007ad, cacdc9fc199e49c38a88c94cd1e5158a; no voters:
I20260619 02:37:52.356952 24070 raft_consensus.cc:2454] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cacdc9fc199e49c38a88c94cd1e5158a in term 0.
I20260619 02:37:52.357105 24404 raft_consensus.cc:2781] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:37:52.357208 24404 raft_consensus.cc:482] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:37:52.357264 24404 raft_consensus.cc:3037] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:52.358347 24404 raft_consensus.cc:504] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:52.358668 24404 leader_election.cc:288] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [CANDIDATE]: Term 1 election: Requested vote from peers 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093), e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627)
I20260619 02:37:52.359052 24070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b220efdee514ab2bfee9271266e585c" candidate_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dae38bb20144c8af479b4a960655ae"
I20260619 02:37:52.359041 24209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b220efdee514ab2bfee9271266e585c" candidate_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad"
I20260619 02:37:52.359172 24070 raft_consensus.cc:3037] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:52.359172 24209 raft_consensus.cc:3037] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:37:52.360466 24070 raft_consensus.cc:2454] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cacdc9fc199e49c38a88c94cd1e5158a in term 1.
I20260619 02:37:52.360466 24209 raft_consensus.cc:2454] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cacdc9fc199e49c38a88c94cd1e5158a in term 1.
I20260619 02:37:52.360793 24274 leader_election.cc:302] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae79ba41e814bff9ea30d9fdd1007ad, cacdc9fc199e49c38a88c94cd1e5158a; no voters:
I20260619 02:37:52.360946 24404 raft_consensus.cc:2781] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:37:52.361178 24404 raft_consensus.cc:686] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 1 LEADER]: Becoming Leader. State: Replica: cacdc9fc199e49c38a88c94cd1e5158a, State: Running, Role: LEADER
I20260619 02:37:52.361578 24404 consensus_queue.cc:237] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:37:52.364755 23929 catalog_manager.cc:5495] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a reported cstate change: term changed from 0 to 1, leader changed from <none> to cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195). New cstate: current_term: 1 leader_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } health_report { overall_health: UNKNOWN } } }
W20260619 02:37:52.374125 24118 tablet.cc:2369] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:37:52.382566 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.378955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.7.194.195:40329)
0619 02:37:52.379094 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:37:52.379099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:37:52.379113 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:37:52.379241 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:37:52.379243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:37:52.379297 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:37:52.379409 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:37:52.379637 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.380141 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.381098 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:37:52.381326 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:37:52.381431 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:37:52.381764 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:37:52.382173 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:37:52.382318 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:37:52.382363 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 02:37:52.382420 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:37:52.388484 24070 raft_consensus.cc:1261] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Refusing update from remote peer cacdc9fc199e49c38a88c94cd1e5158a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:52.388491 24209 raft_consensus.cc:1261] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 FOLLOWER]: Refusing update from remote peer cacdc9fc199e49c38a88c94cd1e5158a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:37:52.389216 24404 consensus_queue.cc:1035] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:52.389442 24407 consensus_queue.cc:1035] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:37:52.405951 23929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:52.406129 23929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:37:52.408550 23929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Sending DeleteTablet for 3 replicas of tablet 5b220efdee514ab2bfee9271266e585c
I20260619 02:37:52.409400 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 23967
I20260619 02:37:52.409413 24321 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b220efdee514ab2bfee9271266e585c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:52 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:47936
I20260619 02:37:52.409428 24189 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b220efdee514ab2bfee9271266e585c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:37:52 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:44276
I20260619 02:37:52.411580 24420 tablet_replica.cc:330] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: stopping tablet replica
I20260619 02:37:52.411916 24420 raft_consensus.cc:2229] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:37:52.412249 24420 raft_consensus.cc:2258] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:52.413014 24421 tablet_replica.cc:330] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: stopping tablet replica
I20260619 02:37:52.413426 24421 raft_consensus.cc:2229] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:37:52.413939 24421 raft_consensus.cc:2258] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:37:52.413987 24420 ts_tablet_manager.cc:1905] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:37:52.415494 24421 ts_tablet_manager.cc:1905] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:37:52.416096 23913 connection.cc:535] client connection to 127.7.194.193:38627 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 02:37:52.416277 23913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:37:52.416785 24420 ts_tablet_manager.cc:1918] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:52.416880 24420 log.cc:1196] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/5b220efdee514ab2bfee9271266e585c
I20260619 02:37:52.417219 24420 ts_tablet_manager.cc:1939] T 5b220efdee514ab2bfee9271266e585c P 4ae79ba41e814bff9ea30d9fdd1007ad: Deleting consensus metadata
I20260619 02:37:52.417374 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 24121
I20260619 02:37:52.419516 24421 ts_tablet_manager.cc:1918] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:37:52.419622 24421 log.cc:1196] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/5b220efdee514ab2bfee9271266e585c
I20260619 02:37:52.419687 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.419391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38627 (local address 127.0.0.1:55282)
0619 02:37:52.419525 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:52.419601 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 02:37:52.420017 24421 ts_tablet_manager.cc:1939] T 5b220efdee514ab2bfee9271266e585c P cacdc9fc199e49c38a88c94cd1e5158a: Deleting consensus metadata
W20260619 02:37:52.419855 23913 catalog_manager.cc:4584] TS e7dae38bb20144c8af479b4a960655ae (127.7.194.193:38627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b220efdee514ab2bfee9271266e585c: Network error: Client connection negotiation failed: client connection to 127.7.194.193:38627: connect: Connection refused (error 111)
I20260619 02:37:52.421139 23913 catalog_manager.cc:4841] TS cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329): tablet 5b220efdee514ab2bfee9271266e585c (table test-table [id=087e1997bbb04c7cbf95b878ce3f9c8e]) successfully deleted
I20260619 02:37:52.424376 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 24258
I20260619 02:37:52.425231 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:37:52.424536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:46093 (local address 127.0.0.1:44286)
0619 02:37:52.425074 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 02:37:52.425137 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.194:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413}
I20260619 02:37:52.430297 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 23896
I20260619 02:38:12.436981 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38615
--webserver_interface=127.7.194.254
--webserver_port=32927
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:12.547837 24423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:12.548097 24423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:12.548161 24423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:12.552099 24423 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:38:12.552152 24423 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:38:12.552196 24423 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:38:12.552217 24423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:12.552242 24423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:12.552277 24423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:12.552313 24423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:12.552341 24423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:12.557407 24423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:12.558421 24423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:12.559355 24423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:12.568367 24428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:12.568459 24429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:12.568459 24431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:12.568588 24423 server_base.cc:1034] running on GCE node
I20260619 02:38:12.569270 24423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:12.570477 24423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:12.571663 24423 hybrid_clock.cc:648] HybridClock initialized: now 1781836692571629 us; error 48 us; skew 500 ppm
Jun 19 02:38:12 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836692, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:12.578753 24423 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:38:12.579676 24423 webserver.cc:460] Webserver started at http://127.7.194.254:32927/ using document root <none> and password file <none>
I20260619 02:38:12.580255 24423 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:12.580345 24423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:12.585067 24423 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:38:12.588093 24438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:12.589730 24423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 02:38:12.589848 24423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "7154e3ae52f34384900e9d44943ea048"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "eb9d5850581a0bf12c41a38c5a536f87"
server_key_iv: "1a7c069e6848430b6923e0e7b2b89944"
server_key_version: "encryptionkey@0"
I20260619 02:38:12.590384 24423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:12.617818 24423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:12.620952 24423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:12.621181 24423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:12.628515 24423 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:38615
I20260619 02:38:12.628803 24490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:38615 every 8 connection(s)
I20260619 02:38:12.629778 24423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:12.633711 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 24423
I20260619 02:38:12.634111 24491 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:38:12.634136 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:38627
--local_ip_for_outbound_sockets=127.7.194.193
--tserver_master_addrs=127.7.194.254:38615
--webserver_port=43683
--webserver_interface=127.7.194.193
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:38:12.635780 24491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Bootstrap starting.
I20260619 02:38:12.660658 24491 log.cc:824] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:12.675001 24491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:38:12.675459 24491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Bootstrap complete.
I20260619 02:38:12.681079 24491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:38:12.681623 24491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7154e3ae52f34384900e9d44943ea048, State: Initialized, Role: FOLLOWER
I20260619 02:38:12.682193 24491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:38:12.682332 24491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:12.682395 24491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:12.682494 24491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:38:12.685047 24491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:38:12.685411 24491 leader_election.cc:302] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7154e3ae52f34384900e9d44943ea048; no voters:
I20260619 02:38:12.685793 24491 leader_election.cc:288] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:38:12.685943 24495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:38:12.686246 24495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [term 3 LEADER]: Becoming Leader. State: Replica: 7154e3ae52f34384900e9d44943ea048, State: Running, Role: LEADER
I20260619 02:38:12.686566 24495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } }
I20260619 02:38:12.687122 24491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:12.688104 24496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7154e3ae52f34384900e9d44943ea048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } } }
I20260619 02:38:12.688231 24496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:12.688081 24497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7154e3ae52f34384900e9d44943ea048. Latest consensus state: current_term: 3 leader_uuid: "7154e3ae52f34384900e9d44943ea048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7154e3ae52f34384900e9d44943ea048" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38615 } } }
I20260619 02:38:12.688591 24497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:12.688908 24504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:12.694593 24504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:12.695295 24504 catalog_manager.cc:1259] Loaded cluster ID: 88a46d67c68e4c288db67f48a703f80b
I20260619 02:38:12.695354 24504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:12.697909 24504 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:12.698642 24504 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Loaded TSK: 0
I20260619 02:38:12.698724 24504 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7154e3ae52f34384900e9d44943ea048: Loaded TSK: 1
F20260619 02:38:12.699518 24504 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc85d3329ad google::LogMessage::Fail() at ??:0
@ 0x7fc85d33588a google::LogMessage::SendToLog() at ??:0
@ 0x7fc85d3324de google::LogMessage::Flush() at ??:0
@ 0x7fc85d333a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc85fff618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc85ffeef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc867c977bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc867c7199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc867c72178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc867cb3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc867cd282b std::function<>::operator()() at ??:0
@ 0x7fc867c71b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc867c72f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc867c6f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc867cb3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc85f5843d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc85f584c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc85f5863f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc85f575512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc860bb96db start_thread at ??:0
@ 0x7fc85be3061f clone at ??:0
W20260619 02:38:12.799355 24493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:12.799618 24493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:12.799665 24493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:12.803887 24493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:12.803969 24493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:12.804060 24493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:38:12.809746 24493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:38627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=43683
--tserver_master_addrs=127.7.194.254:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:12.810846 24493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:12.811914 24493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:12.822841 24538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:12.822883 24541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:12.823202 24539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:12.823312 24493 server_base.cc:1034] running on GCE node
I20260619 02:38:12.823799 24493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:12.824628 24493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:12.825829 24493 hybrid_clock.cc:648] HybridClock initialized: now 1781836692825787 us; error 50 us; skew 500 ppm
Jun 19 02:38:12 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836692, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:12.829162 24493 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:38:12.830142 24493 webserver.cc:460] Webserver started at http://127.7.194.193:43683/ using document root <none> and password file <none>
I20260619 02:38:12.830729 24493 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:12.830790 24493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:12.834954 24493 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:38:12.837430 24548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:12.838850 24493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:38:12.839017 24493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "e7dae38bb20144c8af479b4a960655ae"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "f60eb4e8ccc1ba4a0ab74b811c6bb4b4"
server_key_iv: "77a7d7f2108cb6bd2678ce4d5b7269a1"
server_key_version: "encryptionkey@0"
I20260619 02:38:12.839586 24493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:12.856249 24493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:12.860141 24493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:12.860373 24493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:12.861011 24493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:12.862286 24555 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:38:12.866546 24493 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:38:12.866606 24493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 02:38:12.866655 24493 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:38:12.868355 24493 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:38:12.868420 24493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 02:38:12.868693 24555 tablet_bootstrap.cc:492] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Bootstrap starting.
I20260619 02:38:12.879631 24493 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:38627
I20260619 02:38:12.880921 24493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:38:12.882673 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 24493
I20260619 02:38:12.882031 24662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:38627 every 8 connection(s)
I20260619 02:38:12.883220 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:46093
--local_ip_for_outbound_sockets=127.7.194.194
--tserver_master_addrs=127.7.194.254:38615
--webserver_port=46091
--webserver_interface=127.7.194.194
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:38:12.890983 24555 log.cc:824] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:12.900750 24555 tablet_bootstrap.cc:492] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:38:12.901221 24555 tablet_bootstrap.cc:492] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Bootstrap complete.
I20260619 02:38:12.902693 24555 ts_tablet_manager.cc:1397] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.007s
I20260619 02:38:12.906229 24555 raft_consensus.cc:348] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:38:12.906749 24555 raft_consensus.cc:729] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dae38bb20144c8af479b4a960655ae, State: Initialized, Role: FOLLOWER
I20260619 02:38:12.907284 24555 consensus_queue.cc:260] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:38:12.908382 24555 ts_tablet_manager.cc:1428] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 02:38:12.924427 24666 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:38:12.886093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:46211)
0619 02:38:12.886705 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:12.886764 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:12.887858 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:12.923436 (+ 35578us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":168,"threads_started":1}
W20260619 02:38:12.925001 24663 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:38615 (0 consecutive failures): Network error: Failed to ping master at 127.7.194.254:38615: Client connection negotiation failed: client connection to 127.7.194.254:38615: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:38:12.925820 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:12.925486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:42139)
0619 02:38:12.925677 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:12.925754 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 02:38:12.926499 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:12.926264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:35261)
0619 02:38:12.926393 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:12.926444 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 02:38:12.926680 24663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:38:13.005563 24665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:13.005849 24665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:13.005895 24665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:13.010017 24665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:13.010079 24665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:13.010186 24665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:38:13.015694 24665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:46093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=46091
--tserver_master_addrs=127.7.194.254:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:13.016835 24665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:13.018072 24665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:13.027850 24676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:13.027858 24674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:13.027842 24673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:13.028221 24665 server_base.cc:1034] running on GCE node
I20260619 02:38:13.028656 24665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:13.029335 24665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:13.030537 24665 hybrid_clock.cc:648] HybridClock initialized: now 1781836693030497 us; error 50 us; skew 500 ppm
Jun 19 02:38:13 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836693, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:13.033599 24665 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:38:13.034502 24665 webserver.cc:460] Webserver started at http://127.7.194.194:46091/ using document root <none> and password file <none>
I20260619 02:38:13.035076 24665 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:13.035173 24665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:13.039111 24665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:38:13.041345 24683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:13.042711 24665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:38:13.042860 24665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "4ae79ba41e814bff9ea30d9fdd1007ad"
format_stamp: "Formatted at 2026-06-19 02:37:30 on dist-test-slave-9tkz"
server_key: "354d8b46a6df2a7bfd1295aa9bd055dd"
server_key_iv: "136bf306c95ae4a3b75b996b8f028dd6"
server_key_version: "encryptionkey@0"
I20260619 02:38:13.043519 24665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:13.065377 24665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:13.068624 24665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:13.068861 24665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:13.069517 24665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:13.070475 24665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:13.070554 24665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:13.070623 24665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:13.070673 24665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:13.080207 24665 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:46093
I20260619 02:38:13.080235 24796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:46093 every 8 connection(s)
I20260619 02:38:13.081364 24665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:38:13.082371 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 24665
I20260619 02:38:13.082825 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:40329
--local_ip_for_outbound_sockets=127.7.194.195
--tserver_master_addrs=127.7.194.254:38615
--webserver_port=46317
--webserver_interface=127.7.194.195
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:38:13.085628 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:13.083711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:60417)
0619 02:38:13.084201 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:13.084345 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":154,"threads_started":1}
W20260619 02:38:13.086067 24797 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:38615 (0 consecutive failures): Network error: Failed to ping master at 127.7.194.254:38615: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
I20260619 02:38:13.086817 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:13.086513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:55291)
0619 02:38:13.086707 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:13.086759 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 02:38:13.087582 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:13.087353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:44371)
0619 02:38:13.087490 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:13.087532 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 02:38:13.087821 24797 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:38:13.197345 24799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:13.197724 24799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:13.197832 24799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:13.201946 24799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:13.202016 24799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:13.202081 24799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:38:13.207504 24799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:40329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=46317
--tserver_master_addrs=127.7.194.254:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:13.208750 24799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:13.209923 24799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:13.219672 24806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:13.219712 24808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:13.219777 24805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:13.220118 24799 server_base.cc:1034] running on GCE node
I20260619 02:38:13.220511 24799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:13.221196 24799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:13.222419 24799 hybrid_clock.cc:648] HybridClock initialized: now 1781836693222372 us; error 58 us; skew 500 ppm
Jun 19 02:38:13 dist-test-slave-9tkz krb5kdc[23346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836693, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:13.225503 24799 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:38:13.226493 24799 webserver.cc:460] Webserver started at http://127.7.194.195:46317/ using document root <none> and password file <none>
I20260619 02:38:13.227077 24799 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:13.227169 24799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:13.231197 24799 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 02:38:13.233500 24815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:13.234803 24799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:38:13.234946 24799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "cacdc9fc199e49c38a88c94cd1e5158a"
format_stamp: "Formatted at 2026-06-19 02:37:31 on dist-test-slave-9tkz"
server_key: "9d977c24985fa99117ca2086e0f2474b"
server_key_iv: "0671a60c93e4f3f05f2a72e7259c960e"
server_key_version: "encryptionkey@0"
I20260619 02:38:13.235520 24799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:13.255054 24799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:13.258322 24799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:13.258569 24799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:13.259188 24799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:13.260106 24799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:13.260172 24799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:13.260244 24799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:13.260288 24799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:13.269766 24799 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:40329
I20260619 02:38:13.269847 24928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:40329 every 8 connection(s)
I20260619 02:38:13.270845 24799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:38:13.274852 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:13.272994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:54691)
0619 02:38:13.273506 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:13.273641 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":146,"threads_started":1}
W20260619 02:38:13.275291 24929 heartbeater.cc:644] Failed to heartbeat to 127.7.194.254:38615 (0 consecutive failures): Network error: Failed to ping master at 127.7.194.254:38615: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
I20260619 02:38:13.275954 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:13.275699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:48527)
0619 02:38:13.275820 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:13.275875 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:38:13.276715 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:13.276463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:49139)
0619 02:38:13.276592 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:13.276649 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 02:38:13.276892 24929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:38:13.280278 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 24799
I20260619 02:38:13.927789 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:13.927321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:39635)
0619 02:38:13.927605 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:13.927669 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
I20260619 02:38:14.088800 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:14.088386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:49863)
0619 02:38:14.088638 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:14.088696 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":67,"threads_started":1}
I20260619 02:38:14.277977 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:14.277467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:46521)
0619 02:38:14.277795 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:14.277852 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":167,"threads_started":1}
I20260619 02:38:14.414113 24936 raft_consensus.cc:482] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:14.414376 24936 raft_consensus.cc:504] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:38:14.415540 24936 leader_election.cc:288] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329), 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093)
Jun 19 02:38:14 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836692, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.195@KRBTEST.COM
Jun 19 02:38:14 dist-test-slave-9tkz krb5kdc[23346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836692, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.194@KRBTEST.COM
I20260619 02:38:14.430002 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:14.415826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:40329 (local address 127.7.194.193:44559)
0619 02:38:14.415979 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:14.415993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:14.416133 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:14.418473 (+ 2340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:14.418478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:14.418879 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:14.419388 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:14.419402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:14.420668 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:14.420672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:14.421639 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:14.421648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:14.422353 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:14.422356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:14.422586 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:14.423347 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:14.423390 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:14.425060 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:14.427488 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:14.427496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:14.427512 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:14.427843 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:14.428204 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:14.428206 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:14.428207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:14.428328 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:14.429115 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:14.429120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:14.429321 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 02:38:14.429470 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 02:38:14.429786 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":117}
I20260619 02:38:14.430059 24937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:14.416031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:44559 (local address 127.7.194.195:40329)
0619 02:38:14.416437 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:14.416461 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:14.416531 (+ 70us) server_negotiation.cc:407] Connection header received
0619 02:38:14.417506 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:14.417516 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:14.418024 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:14.418372 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:14.419562 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:14.420531 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:14.421762 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:14.422228 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:14.422405 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:14.425311 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:14.425318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:14.425338 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:14.425376 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:14.427363 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:14.427987 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:14.427991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:14.427997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:14.428122 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:14.428445 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:14.428447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:14.428448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:14.429006 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:14.429140 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:14.429638 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 02:38:14.429884 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":76,"threads_started":1}
I20260619 02:38:14.430460 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:14.416313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:45865 (local address 127.7.194.194:46093)
0619 02:38:14.416654 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:14.416676 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:14.416709 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:38:14.417523 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:14.417530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:14.418024 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:14.418372 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:14.419560 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:14.420615 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:14.421768 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:14.422227 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:14.422415 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:14.425650 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:14.425655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:14.425666 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:14.425694 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:14.427466 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:14.427945 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:14.427953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:14.427960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:14.428113 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:14.428550 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:14.428553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:14.428554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:14.429006 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:14.429141 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:14.429978 (+ 837us) server_negotiation.cc:299] Negotiation successful
0619 02:38:14.430245 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":132,"threads_started":1}
I20260619 02:38:14.430151 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:14.416093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:46093 (local address 127.7.194.193:45865)
0619 02:38:14.416333 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:14.416346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:14.416485 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:14.418875 (+ 2390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:14.418878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:14.419037 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:14.419387 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:14.419402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:14.420744 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:14.420749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:14.421638 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:14.421653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:14.422360 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:14.422364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:14.422584 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:14.423993 (+ 1409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:14.424010 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:14.425424 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:14.427578 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:14.427580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:14.427581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:14.427827 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:14.428415 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:14.428416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:14.428418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:14.428471 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:14.429649 (+ 1178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:14.429652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:14.429734 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 02:38:14.429841 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 02:38:14.429952 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":83,"thread_start_us":53,"threads_started":1}
W20260619 02:38:14.432729 24549 leader_election.cc:341] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093): Not found: Tablet not found: 5b220efdee514ab2bfee9271266e585c
W20260619 02:38:14.432973 24549 leader_election.cc:341] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329): Not found: Tablet not found: 5b220efdee514ab2bfee9271266e585c
I20260619 02:38:14.433059 24549 leader_election.cc:302] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7dae38bb20144c8af479b4a960655ae; no voters: 4ae79ba41e814bff9ea30d9fdd1007ad, cacdc9fc199e49c38a88c94cd1e5158a
I20260619 02:38:14.433333 24936 raft_consensus.cc:2726] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:38:14.928895 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:14.928573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:56141)
0619 02:38:14.928734 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:14.928806 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 02:38:15.090093 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:15.089642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:55671)
0619 02:38:15.089926 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:15.089997 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260619 02:38:15.279214 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:15.278810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:34267)
0619 02:38:15.279071 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:15.279124 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20260619 02:38:15.930248 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:15.929823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:56717)
0619 02:38:15.930086 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:15.930147 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260619 02:38:16.091310 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:16.090896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:54025)
0619 02:38:16.091151 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:16.091206 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":68,"threads_started":1}
I20260619 02:38:16.280618 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:16.280024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:58931)
0619 02:38:16.280282 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:16.280350 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260619 02:38:16.560952 24945 raft_consensus.cc:482] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:16.561149 24945 raft_consensus.cc:504] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:38:16.561719 24945 leader_election.cc:288] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329), 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093)
W20260619 02:38:16.562543 24549 leader_election.cc:341] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329): Not found: Tablet not found: 5b220efdee514ab2bfee9271266e585c
W20260619 02:38:16.562695 24549 leader_election.cc:341] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093): Not found: Tablet not found: 5b220efdee514ab2bfee9271266e585c
I20260619 02:38:16.562755 24549 leader_election.cc:302] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7dae38bb20144c8af479b4a960655ae; no voters: 4ae79ba41e814bff9ea30d9fdd1007ad, cacdc9fc199e49c38a88c94cd1e5158a
I20260619 02:38:16.562958 24945 raft_consensus.cc:2726] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:38:16.931501 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:16.931055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:50443)
0619 02:38:16.931335 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:16.931399 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260619 02:38:17.092604 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:17.092154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:40933)
0619 02:38:17.092445 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:17.092508 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
I20260619 02:38:17.281965 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:17.281475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:38361)
0619 02:38:17.281781 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:17.281862 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":129,"threads_started":1}
I20260619 02:38:17.932955 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:17.932490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:54537)
0619 02:38:17.932778 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:17.932835 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":83,"threads_started":1}
I20260619 02:38:18.094049 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:18.093601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:41147)
0619 02:38:18.093882 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:18.093955 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260619 02:38:18.262635 24951 raft_consensus.cc:482] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:18.262777 24951 raft_consensus.cc:504] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:38:18.263247 24951 leader_election.cc:288] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329), 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093)
W20260619 02:38:18.264050 24549 leader_election.cc:341] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329): Not found: Tablet not found: 5b220efdee514ab2bfee9271266e585c
W20260619 02:38:18.264210 24549 leader_election.cc:341] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093): Not found: Tablet not found: 5b220efdee514ab2bfee9271266e585c
I20260619 02:38:18.264273 24549 leader_election.cc:302] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7dae38bb20144c8af479b4a960655ae; no voters: 4ae79ba41e814bff9ea30d9fdd1007ad, cacdc9fc199e49c38a88c94cd1e5158a
I20260619 02:38:18.264453 24951 raft_consensus.cc:2726] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:38:18.283149 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:18.282724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:55455)
0619 02:38:18.282982 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:18.283051 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260619 02:38:18.934168 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:18.933741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:40197)
0619 02:38:18.934015 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:18.934070 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":72,"threads_started":1}
I20260619 02:38:19.095389 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:19.094886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:41803)
0619 02:38:19.095211 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:19.095283 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260619 02:38:19.284379 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:19.283950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:44467)
0619 02:38:19.284228 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:19.284284 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260619 02:38:19.935527 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:19.934960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:39419)
0619 02:38:19.935247 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:19.935380 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":92,"threads_started":1}
I20260619 02:38:20.096545 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:20.096120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:50131)
0619 02:38:20.096394 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:20.096449 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":71,"threads_started":1}
I20260619 02:38:20.285756 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:20.285206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:50507)
0619 02:38:20.285530 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:20.285655 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":99,"threads_started":1}
I20260619 02:38:20.788911 24959 raft_consensus.cc:482] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:20.789042 24959 raft_consensus.cc:504] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacdc9fc199e49c38a88c94cd1e5158a" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 40329 } } peers { permanent_uuid: "4ae79ba41e814bff9ea30d9fdd1007ad" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 46093 } } peers { permanent_uuid: "e7dae38bb20144c8af479b4a960655ae" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38627 } }
I20260619 02:38:20.789637 24959 leader_election.cc:288] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329), 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093)
W20260619 02:38:20.790548 24549 leader_election.cc:341] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cacdc9fc199e49c38a88c94cd1e5158a (127.7.194.195:40329): Not found: Tablet not found: 5b220efdee514ab2bfee9271266e585c
W20260619 02:38:20.790688 24549 leader_election.cc:341] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ae79ba41e814bff9ea30d9fdd1007ad (127.7.194.194:46093): Not found: Tablet not found: 5b220efdee514ab2bfee9271266e585c
I20260619 02:38:20.790756 24549 leader_election.cc:302] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7dae38bb20144c8af479b4a960655ae; no voters: 4ae79ba41e814bff9ea30d9fdd1007ad, cacdc9fc199e49c38a88c94cd1e5158a
I20260619 02:38:20.790936 24959 raft_consensus.cc:2726] T 5b220efdee514ab2bfee9271266e585c P e7dae38bb20144c8af479b4a960655ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:38:20.936775 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:20.936291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:45949)
0619 02:38:20.936596 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:20.936662 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
I20260619 02:38:21.097790 24961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:21.097281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:35429)
0619 02:38:21.097634 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:21.097694 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
I20260619 02:38:21.286968 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:21.286521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:59737)
0619 02:38:21.286804 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:21.286861 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260619 02:38:21.938005 24963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:21.937524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:44721)
0619 02:38:21.937844 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:21.937905 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
I20260619 02:38:22.099078 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:22.098600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:52569)
0619 02:38:22.098892 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:22.098972 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20260619 02:38:22.288597 24965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:22.287975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:58417)
0619 02:38:22.288420 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:22.288479 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":102,"threads_started":1}
I20260619 02:38:22.939452 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:22.938963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:58979)
0619 02:38:22.939242 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:22.939311 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260619 02:38:23.100461 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:23.100016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:50469)
0619 02:38:23.100294 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:23.100365 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260619 02:38:23.289978 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:23.289487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:45923)
0619 02:38:23.289818 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:23.289881 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":152,"threads_started":1}
I20260619 02:38:23.940793 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:23.940355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:43273)
0619 02:38:23.940626 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:23.940689 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260619 02:38:24.101686 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:24.101196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:38187)
0619 02:38:24.101451 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:24.101517 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260619 02:38:24.291139 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:24.290745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:57317)
0619 02:38:24.290984 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:24.291042 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260619 02:38:24.942030 24972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:24.941614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:43613)
0619 02:38:24.941867 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:24.941927 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260619 02:38:25.102906 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:25.102509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:58453)
0619 02:38:25.102750 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:25.102812 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260619 02:38:25.292562 24974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:25.292052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:55431)
0619 02:38:25.292379 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:25.292453 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260619 02:38:25.943140 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:25.942708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:32971)
0619 02:38:25.942987 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:25.943040 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260619 02:38:26.104171 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:26.103738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:43993)
0619 02:38:26.104016 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:26.104076 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":73,"threads_started":1}
I20260619 02:38:26.293854 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:26.293377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:49123)
0619 02:38:26.293692 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:26.293749 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":94,"threads_started":1}
I20260619 02:38:26.944428 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:26.943980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:46239)
0619 02:38:26.944267 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:26.944328 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260619 02:38:27.105432 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:27.105012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:36017)
0619 02:38:27.105276 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:27.105336 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260619 02:38:27.295110 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:27.294646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:55659)
0619 02:38:27.294897 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:27.294961 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260619 02:38:27.945914 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:27.945452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.193:38233)
0619 02:38:27.945742 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:27.945812 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":89,"threads_started":1}
I20260619 02:38:28.106791 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:28.106393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.194:55363)
0619 02:38:28.106625 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:28.106680 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260619 02:38:28.289224 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 24493
I20260619 02:38:28.295259 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 24665
I20260619 02:38:28.296814 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:28.296423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38615 (local address 127.7.194.195:37095)
0619 02:38:28.296669 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:28.296720 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.254:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260619 02:38:28.301076 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 24799
2026-06-19T02:38:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59314 ms)
[----------] 30 tests from SecurityITest (244026 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:28 dist-test-slave-9tkz krb5kdc[24986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:28 dist-test-slave-9tkz krb5kdc[24986](info): set up 2 sockets
Jun 19 02:38:28 dist-test-slave-9tkz krb5kdc[24986](info): commencing operation
krb5kdc: starting...
W20260619 02:38:29.342283 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:29 dist-test-slave-9tkz krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:38:29.468479 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:33351
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:33351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:29.580963 24999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:29.581311 24999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:29.581379 24999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:29.585464 24999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:29.585588 24999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:29.585647 24999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:29.585707 24999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:29.585747 24999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:29.591078 24999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:33351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:33351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:29.592247 24999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:29.593345 24999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:29.602131 25004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:29.602276 25007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:29.602342 25005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:29.602418 24999 server_base.cc:1034] running on GCE node
I20260619 02:38:29.603053 24999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:29.604178 24999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:29.605356 24999 hybrid_clock.cc:648] HybridClock initialized: now 1781836709605326 us; error 41 us; skew 500 ppm
Jun 19 02:38:29 dist-test-slave-9tkz krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:29.608078 24999 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:38:29.609020 24999 webserver.cc:460] Webserver started at http://127.7.194.254:42295/ using document root <none> and password file <none>
I20260619 02:38:29.609637 24999 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:29.609728 24999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:29.609963 24999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:29.612000 24999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "3c4a17f4d07740dca368697b304e7342"
format_stamp: "Formatted at 2026-06-19 02:38:29 on dist-test-slave-9tkz"
server_key: "8b3f74ef29f70db579eca31be9d6a73a"
server_key_iv: "66b1ffd8e3155d1c71f5fd2f4c414e84"
server_key_version: "encryptionkey@0"
I20260619 02:38:29.612561 24999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "3c4a17f4d07740dca368697b304e7342"
format_stamp: "Formatted at 2026-06-19 02:38:29 on dist-test-slave-9tkz"
server_key: "8b3f74ef29f70db579eca31be9d6a73a"
server_key_iv: "66b1ffd8e3155d1c71f5fd2f4c414e84"
server_key_version: "encryptionkey@0"
I20260619 02:38:29.616300 24999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:38:29.618963 25014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:29.620208 24999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:38:29.620347 24999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "3c4a17f4d07740dca368697b304e7342"
format_stamp: "Formatted at 2026-06-19 02:38:29 on dist-test-slave-9tkz"
server_key: "8b3f74ef29f70db579eca31be9d6a73a"
server_key_iv: "66b1ffd8e3155d1c71f5fd2f4c414e84"
server_key_version: "encryptionkey@0"
I20260619 02:38:29.620462 24999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:29.643131 24999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:29.646294 24999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:29.646519 24999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:29.653646 24999 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:33351
I20260619 02:38:29.653666 25066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:33351 every 8 connection(s)
I20260619 02:38:29.654605 24999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:29.655084 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 24999
I20260619 02:38:29.655258 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:29.655591 7947 external_mini_cluster.cc:1428] Setting key a1155ec503dd279f53c68931c3fc8d10
I20260619 02:38:29.657912 25067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:29.663362 25067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342: Bootstrap starting.
Jun 19 02:38:29 dist-test-slave-9tkz krb5kdc[24986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:29.665828 25067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:29.666669 25067 log.cc:824] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:29.668893 25067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342: No bootstrap required, opened a new log
I20260619 02:38:29.670146 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:29.656997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.7.194.254:33351)
0619 02:38:29.657458 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:29.657473 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:29.657508 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:38:29.658257 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:29.658275 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:29.658623 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:29.658971 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:29.659808 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:29.660782 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:29.661722 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:29.662182 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:29.662354 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:29.665089 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:29.665094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:29.665107 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:29.665149 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:29.667045 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:29.667523 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:29.667528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:29.667532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:29.667639 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:29.667931 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:29.667933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:29.667934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:29.668429 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:29.668582 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:29.668881 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 02:38:29.669172 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":163,"threads_started":1}
I20260619 02:38:29.672462 25067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a17f4d07740dca368697b304e7342" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33351 } }
I20260619 02:38:29.672693 25067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:29.672777 25067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4a17f4d07740dca368697b304e7342, State: Initialized, Role: FOLLOWER
I20260619 02:38:29.673177 25067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a17f4d07740dca368697b304e7342" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33351 } }
I20260619 02:38:29.673318 25067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:29.673403 25067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:29.673527 25067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:29.674479 25067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a17f4d07740dca368697b304e7342" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33351 } }
I20260619 02:38:29.674817 25067 leader_election.cc:302] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c4a17f4d07740dca368697b304e7342; no voters:
I20260619 02:38:29.675133 25067 leader_election.cc:288] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:29.675298 25072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:29.675522 25072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4a17f4d07740dca368697b304e7342, State: Running, Role: LEADER
I20260619 02:38:29.675848 25072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a17f4d07740dca368697b304e7342" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33351 } }
I20260619 02:38:29.676389 25067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:29.677397 25073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c4a17f4d07740dca368697b304e7342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a17f4d07740dca368697b304e7342" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33351 } } }
I20260619 02:38:29.677517 25073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:29.677862 25074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c4a17f4d07740dca368697b304e7342. Latest consensus state: current_term: 1 leader_uuid: "3c4a17f4d07740dca368697b304e7342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a17f4d07740dca368697b304e7342" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33351 } } }
I20260619 02:38:29.677953 25081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:29.678020 25074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:29.681408 25081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:29.687109 25081 catalog_manager.cc:1347] Generated new cluster ID: 9a30a06f52a74a8583e21915cd055d3c
I20260619 02:38:29.687179 25081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:29.715149 25081 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:29.716560 25081 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:29.724449 25081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342: Generated new TSK 0
I20260619 02:38:29.725297 25081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:38:29.765911 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:33351
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:29.878136 25093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:29.878432 25093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:29.878504 25093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:29.882701 25093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:29.882792 25093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:29.882884 25093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:38:29.888232 25093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:33351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:29.889355 25093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:29.890466 25093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:29.900220 25098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:29.900197 25099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:29.900197 25101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:29.900480 25093 server_base.cc:1034] running on GCE node
I20260619 02:38:29.900942 25093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:29.901652 25093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:29.902812 25093 hybrid_clock.cc:648] HybridClock initialized: now 1781836709902783 us; error 38 us; skew 500 ppm
Jun 19 02:38:29 dist-test-slave-9tkz krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:29.905726 25093 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:38:29.906651 25093 webserver.cc:460] Webserver started at http://127.7.194.193:35287/ using document root <none> and password file <none>
I20260619 02:38:29.907215 25093 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:29.907294 25093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:29.907500 25093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:29.909616 25093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "8a5582aa846e494b8a2da3aedeaf31a1"
format_stamp: "Formatted at 2026-06-19 02:38:29 on dist-test-slave-9tkz"
server_key: "ea4bbe171948a4c673529e2d7c3d7f17"
server_key_iv: "14aad3f490d756d0432a944cd0cd26eb"
server_key_version: "encryptionkey@0"
I20260619 02:38:29.910183 25093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "8a5582aa846e494b8a2da3aedeaf31a1"
format_stamp: "Formatted at 2026-06-19 02:38:29 on dist-test-slave-9tkz"
server_key: "ea4bbe171948a4c673529e2d7c3d7f17"
server_key_iv: "14aad3f490d756d0432a944cd0cd26eb"
server_key_version: "encryptionkey@0"
I20260619 02:38:29.913834 25093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:38:29.916213 25108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:29.917543 25093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:29.917722 25093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "8a5582aa846e494b8a2da3aedeaf31a1"
format_stamp: "Formatted at 2026-06-19 02:38:29 on dist-test-slave-9tkz"
server_key: "ea4bbe171948a4c673529e2d7c3d7f17"
server_key_iv: "14aad3f490d756d0432a944cd0cd26eb"
server_key_version: "encryptionkey@0"
I20260619 02:38:29.917829 25093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:29.942286 25093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:29.945885 25093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:29.946107 25093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:29.946741 25093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:29.947602 25093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:29.947676 25093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:29.947758 25093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:29.947800 25093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:29.956813 25093 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:45045
I20260619 02:38:29.956841 25221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:45045 every 8 connection(s)
I20260619 02:38:29.957756 25093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:38:29.962749 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 25093
I20260619 02:38:29.962885 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:38:29.963233 7947 external_mini_cluster.cc:1428] Setting key c061943d33628eec5978b4075617553d
Jun 19 02:38:29 dist-test-slave-9tkz krb5kdc[24986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:29.972503 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:29.959645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:39043 (local address 127.7.194.254:33351)
0619 02:38:29.959798 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:29.959801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:29.960648 (+ 847us) server_negotiation.cc:407] Connection header received
0619 02:38:29.961531 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:29.961534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:29.961643 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:29.961721 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:29.963128 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:29.963776 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:29.965253 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:29.965475 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:29.965612 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:29.968624 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:29.968627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:29.968629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:29.968652 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:29.970189 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:29.970880 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:29.970883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:29.970884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:29.970956 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:29.971366 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:29.971369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:29.971370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:29.971539 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:29.971610 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:29.972273 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 02:38:29.972382 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:38:29.973793 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:29.960037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:33351 (local address 127.7.194.193:39043)
0619 02:38:29.960487 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:29.960530 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:29.961323 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:29.961901 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:29.961907 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:29.962277 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:29.962957 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:29.962979 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:29.963914 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:29.963918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:29.965108 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:29.965121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:29.965666 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:29.965673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:29.965924 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:29.966602 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:29.966619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:29.968458 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:29.970335 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:29.970343 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:29.970360 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:29.970776 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:29.971057 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:29.971062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:29.971067 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:29.971266 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:29.971641 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:29.971649 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:29.971997 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 02:38:29.972228 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 02:38:29.972597 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260619 02:38:29.975169 25222 heartbeater.cc:347] Connected to a master server at 127.7.194.254:33351
I20260619 02:38:29.975404 25222 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:29.975943 25222 heartbeater.cc:511] Master 127.7.194.254:33351 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:38:29.977619 25031 ts_manager.cc:194] Registered new tserver with Master: 8a5582aa846e494b8a2da3aedeaf31a1 (127.7.194.193:45045)
I20260619 02:38:29.979336 25031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:39043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:38:29.991892 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:33351
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:30.103953 25227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:30.104243 25227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:30.104287 25227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:30.108489 25227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:30.108547 25227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:30.108635 25227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:38:30.114063 25227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:33351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:30.115147 25227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:30.116184 25227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:30.126259 25232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:30.126260 25233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:30.126346 25227 server_base.cc:1034] running on GCE node
W20260619 02:38:30.126260 25235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:30.126994 25227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:30.127725 25227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:30.128898 25227 hybrid_clock.cc:648] HybridClock initialized: now 1781836710128837 us; error 74 us; skew 500 ppm
Jun 19 02:38:30 dist-test-slave-9tkz krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836710, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:30.131938 25227 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:38:30.132889 25227 webserver.cc:460] Webserver started at http://127.7.194.194:39215/ using document root <none> and password file <none>
I20260619 02:38:30.133425 25227 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:30.133488 25227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:30.133731 25227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:30.135771 25227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "33ec1043f0bf44e69fe14c8f338bbdde"
format_stamp: "Formatted at 2026-06-19 02:38:30 on dist-test-slave-9tkz"
server_key: "05059c15a051dbb4609372f7d40ce66f"
server_key_iv: "8804963282e139a1abff0a8632335283"
server_key_version: "encryptionkey@0"
I20260619 02:38:30.136307 25227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "33ec1043f0bf44e69fe14c8f338bbdde"
format_stamp: "Formatted at 2026-06-19 02:38:30 on dist-test-slave-9tkz"
server_key: "05059c15a051dbb4609372f7d40ce66f"
server_key_iv: "8804963282e139a1abff0a8632335283"
server_key_version: "encryptionkey@0"
I20260619 02:38:30.140136 25227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 02:38:30.142693 25242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:30.143957 25227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:30.144104 25227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "33ec1043f0bf44e69fe14c8f338bbdde"
format_stamp: "Formatted at 2026-06-19 02:38:30 on dist-test-slave-9tkz"
server_key: "05059c15a051dbb4609372f7d40ce66f"
server_key_iv: "8804963282e139a1abff0a8632335283"
server_key_version: "encryptionkey@0"
I20260619 02:38:30.144246 25227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:30.157756 25227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:30.160892 25227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:30.161092 25227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:30.161774 25227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:30.162595 25227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:30.162642 25227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:30.162683 25227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:30.162745 25227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:30.172137 25227 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:44311
I20260619 02:38:30.172166 25355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:44311 every 8 connection(s)
I20260619 02:38:30.173079 25227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:38:30.178519 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 25227
I20260619 02:38:30.178656 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:38:30.179029 7947 external_mini_cluster.cc:1428] Setting key 2f2fb63f8a7bf19e4ab958ddfe26cc45
Jun 19 02:38:30 dist-test-slave-9tkz krb5kdc[24986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836710, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:30.188817 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.174998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:56103 (local address 127.7.194.254:33351)
0619 02:38:30.175160 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.175164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.175993 (+ 829us) server_negotiation.cc:407] Connection header received
0619 02:38:30.177014 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.177018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.177074 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.177169 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:30.178514 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.179335 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.181011 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.181403 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.181575 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.184373 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:30.184380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:30.184386 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:30.184433 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:30.186477 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:30.187025 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:30.187031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:30.187035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:30.187157 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:30.187499 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:30.187504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:30.187507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:30.187780 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:30.187904 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.188453 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.188597 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:38:30.189627 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.175383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:33351 (local address 127.7.194.194:56103)
0619 02:38:30.175829 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:30.175872 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:30.176812 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:30.177407 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:30.177417 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:30.177849 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:30.178348 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.178362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.179483 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.179488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.180877 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.180886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.181606 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.181611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.181764 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.182402 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:30.182418 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:30.184179 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:30.186608 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:30.186614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:30.186637 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:30.186917 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:30.187262 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:30.187264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:30.187265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:30.187396 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:30.187917 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:30.187922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:30.188184 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 02:38:30.188355 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 02:38:30.188645 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:38:30.191085 25356 heartbeater.cc:347] Connected to a master server at 127.7.194.254:33351
I20260619 02:38:30.191365 25356 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:30.191944 25356 heartbeater.cc:511] Master 127.7.194.254:33351 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
I20260619 02:38:30.193358 25031 ts_manager.cc:194] Registered new tserver with Master: 33ec1043f0bf44e69fe14c8f338bbdde (127.7.194.194:44311)
I20260619 02:38:30.194139 25031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:56103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:38:30.208334 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:33351
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:30.319456 25361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:30.319810 25361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:30.319909 25361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:30.324543 25361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:30.324632 25361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:30.324720 25361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:38:30.330329 25361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:33351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:30.331459 25361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:30.332531 25361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:30.342345 25366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:30.342429 25369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:30.342372 25361 server_base.cc:1034] running on GCE node
W20260619 02:38:30.342381 25367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:30.343072 25361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:30.343924 25361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:30.345119 25361 hybrid_clock.cc:648] HybridClock initialized: now 1781836710345104 us; error 34 us; skew 500 ppm
Jun 19 02:38:30 dist-test-slave-9tkz krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836710, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:30.348244 25361 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:38:30.349110 25361 webserver.cc:460] Webserver started at http://127.7.194.195:34757/ using document root <none> and password file <none>
I20260619 02:38:30.349756 25361 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:30.349846 25361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:30.350090 25361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:30.352137 25361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "e03dd946636c462f9450ddac932fb4a2"
format_stamp: "Formatted at 2026-06-19 02:38:30 on dist-test-slave-9tkz"
server_key: "8d4afa5898f7346ac45495f1de33a5d1"
server_key_iv: "aeb5787aaeaa49e658224da5e308773d"
server_key_version: "encryptionkey@0"
I20260619 02:38:30.352787 25361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "e03dd946636c462f9450ddac932fb4a2"
format_stamp: "Formatted at 2026-06-19 02:38:30 on dist-test-slave-9tkz"
server_key: "8d4afa5898f7346ac45495f1de33a5d1"
server_key_iv: "aeb5787aaeaa49e658224da5e308773d"
server_key_version: "encryptionkey@0"
I20260619 02:38:30.356685 25361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 02:38:30.359256 25376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:30.360481 25361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:38:30.360613 25361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "e03dd946636c462f9450ddac932fb4a2"
format_stamp: "Formatted at 2026-06-19 02:38:30 on dist-test-slave-9tkz"
server_key: "8d4afa5898f7346ac45495f1de33a5d1"
server_key_iv: "aeb5787aaeaa49e658224da5e308773d"
server_key_version: "encryptionkey@0"
I20260619 02:38:30.360736 25361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:30.382387 25361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:30.385373 25361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:30.385607 25361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:30.386234 25361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:30.387070 25361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:30.387141 25361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:30.387207 25361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:30.387253 25361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:30.396013 25361 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:42933
I20260619 02:38:30.396082 25489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:42933 every 8 connection(s)
I20260619 02:38:30.396951 25361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:38:30.404963 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 25361
I20260619 02:38:30.405109 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:38:30.405503 7947 external_mini_cluster.cc:1428] Setting key a760d072b2dd1e40ee7ebfdbf4198ffb
Jun 19 02:38:30 dist-test-slave-9tkz krb5kdc[24986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836710, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:30.410980 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.398800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:32793 (local address 127.7.194.254:33351)
0619 02:38:30.398938 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.398948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.399862 (+ 914us) server_negotiation.cc:407] Connection header received
0619 02:38:30.400838 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.400842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.400914 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.401019 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:30.402279 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.402820 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.403969 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.404198 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.404308 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.407343 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:30.407346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:30.407349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:30.407378 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:30.408878 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:30.409402 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:30.409406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:30.409408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:30.409493 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:30.409872 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:30.409877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:30.409879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:30.410100 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:30.410206 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.410700 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.410824 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:38:30.411820 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.399259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:33351 (local address 127.7.194.195:32793)
0619 02:38:30.399715 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:30.399757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:30.400619 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:30.401208 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:30.401215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:30.401639 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:30.402127 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.402150 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.402946 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.402949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.403843 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.403852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.404350 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.404354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.404520 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.405365 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:30.405387 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:30.407190 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:30.409011 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:30.409017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:30.409033 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:30.409299 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:30.409623 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:30.409626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:30.409628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:30.409776 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:30.410204 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:30.410211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:30.410464 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 02:38:30.410648 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 02:38:30.410986 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260619 02:38:30.412863 25490 heartbeater.cc:347] Connected to a master server at 127.7.194.254:33351
I20260619 02:38:30.413072 25490 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:30.413527 25490 heartbeater.cc:511] Master 127.7.194.254:33351 requested a full tablet report, sending...
I20260619 02:38:30.414518 25031 ts_manager.cc:194] Registered new tserver with Master: e03dd946636c462f9450ddac932fb4a2 (127.7.194.195:42933)
I20260619 02:38:30.415050 25031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:32793
I20260619 02:38:30.419616 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:38:30.429412 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.421690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.7.194.254:33351)
0619 02:38:30.421792 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.421796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.422043 (+ 247us) server_negotiation.cc:407] Connection header received
0619 02:38:30.422137 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.422139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.422179 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.422238 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:30.423095 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.423715 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.424704 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.424969 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.425067 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.426627 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:30.426631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:30.426633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:30.426654 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:30.427971 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:30.428405 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:30.428407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:30.428408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:30.428480 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:30.428718 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:30.428720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:30.428721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:30.428923 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:30.429025 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.429181 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.429293 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:38:30.432281 25031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:38:30.435503 25031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:38:30.449858 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.444487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:42933 (local address 127.0.0.1:53670)
0619 02:38:30.444899 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:30.444911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:30.445168 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:30.445741 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:30.445747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:30.445769 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:30.445907 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.445914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.446986 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.446989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.448403 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.448414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.449285 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.449288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.449388 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.449428 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 02:38:30.449563 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 02:38:30.449682 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":72,"spinlock_wait_cycles":357120,"thread_start_us":105,"threads_started":1}
I20260619 02:38:30.449874 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.444731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.7.194.195:42933)
0619 02:38:30.445187 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.445204 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.445239 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:38:30.445318 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.445326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.445533 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.445735 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:30.446012 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.446845 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.448531 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.449163 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.449271 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.449419 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.449611 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.449724 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":154,"threads_started":1}
I20260619 02:38:30.450883 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.443765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:44311 (local address 127.0.0.1:43298)
0619 02:38:30.444551 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:30.444607 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:30.445163 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:30.445977 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:30.445980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:30.445998 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:30.446111 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.446118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.447537 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.447540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.448961 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.448967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.450325 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.450329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.450472 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.450534 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 02:38:30.450617 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:38:30.450736 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":171,"thread_start_us":79,"threads_started":1}
I20260619 02:38:30.451109 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.444055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.7.194.193:45045)
0619 02:38:30.444550 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.444566 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.444958 (+ 392us) server_negotiation.cc:407] Connection header received
0619 02:38:30.445477 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.445482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.445657 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.445805 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:30.446161 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.447378 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.449100 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.449991 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.450103 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.450267 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.450856 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.450956 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":64,"threads_started":1}
I20260619 02:38:30.451561 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.443879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.7.194.194:44311)
0619 02:38:30.444460 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.444474 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.444740 (+ 266us) server_negotiation.cc:407] Connection header received
0619 02:38:30.445418 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.445426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.445651 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.445805 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:30.446243 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.447407 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.449081 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.450181 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.450309 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.450449 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.451378 (+ 929us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.451457 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":98,"threads_started":1}
I20260619 02:38:30.451964 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.443791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:45045 (local address 127.0.0.1:38650)
0619 02:38:30.444847 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:30.444863 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:30.445310 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:30.445905 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:30.445908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:30.445922 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:30.446026 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.446032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.447537 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.447540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.448948 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.448959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.450325 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.450329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.450468 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.450534 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 02:38:30.450617 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:38:30.450731 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":225,"spinlock_wait_cycles":633600,"thread_start_us":93,"threads_started":1}
I20260619 02:38:30.453385 25424 tablet_service.cc:1467] Processing CreateTablet for tablet 66287d8a039c4585965928485b0c7913 (DEFAULT_TABLE table=test-table [id=f7baf0acdefa4ddc932a17d79074a1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:30.454479 25424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66287d8a039c4585965928485b0c7913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:30.454384 25290 tablet_service.cc:1467] Processing CreateTablet for tablet 66287d8a039c4585965928485b0c7913 (DEFAULT_TABLE table=test-table [id=f7baf0acdefa4ddc932a17d79074a1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:30.454512 25156 tablet_service.cc:1467] Processing CreateTablet for tablet 66287d8a039c4585965928485b0c7913 (DEFAULT_TABLE table=test-table [id=f7baf0acdefa4ddc932a17d79074a1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:30.455407 25156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66287d8a039c4585965928485b0c7913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:30.455407 25290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66287d8a039c4585965928485b0c7913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:30.461087 25507 tablet_bootstrap.cc:492] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: Bootstrap starting.
I20260619 02:38:30.462685 25508 tablet_bootstrap.cc:492] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1: Bootstrap starting.
I20260619 02:38:30.463407 25509 tablet_bootstrap.cc:492] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: Bootstrap starting.
I20260619 02:38:30.463390 25507 tablet_bootstrap.cc:654] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:30.464749 25507 log.cc:824] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:30.465131 25508 tablet_bootstrap.cc:654] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:30.465871 25509 tablet_bootstrap.cc:654] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:30.466466 25508 log.cc:824] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:30.466825 25509 log.cc:824] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:30.467132 25507 tablet_bootstrap.cc:492] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: No bootstrap required, opened a new log
I20260619 02:38:30.467382 25507 ts_tablet_manager.cc:1397] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:38:30.468214 25508 tablet_bootstrap.cc:492] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1: No bootstrap required, opened a new log
I20260619 02:38:30.468400 25508 ts_tablet_manager.cc:1397] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:38:30.470805 25507 raft_consensus.cc:348] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } }
I20260619 02:38:30.471095 25507 raft_consensus.cc:374] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:30.471184 25507 raft_consensus.cc:729] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e03dd946636c462f9450ddac932fb4a2, State: Initialized, Role: FOLLOWER
I20260619 02:38:30.471385 25509 tablet_bootstrap.cc:492] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: No bootstrap required, opened a new log
I20260619 02:38:30.471619 25509 ts_tablet_manager.cc:1397] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 02:38:30.471558 25508 raft_consensus.cc:348] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } }
I20260619 02:38:30.471662 25507 consensus_queue.cc:260] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } }
I20260619 02:38:30.471825 25508 raft_consensus.cc:374] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:30.471905 25508 raft_consensus.cc:729] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5582aa846e494b8a2da3aedeaf31a1, State: Initialized, Role: FOLLOWER
I20260619 02:38:30.472370 25508 consensus_queue.cc:260] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } }
I20260619 02:38:30.472664 25507 ts_tablet_manager.cc:1428] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:38:30.473096 25490 heartbeater.cc:503] Master 127.7.194.254:33351 was elected leader, sending a full tablet report...
I20260619 02:38:30.473366 25508 ts_tablet_manager.cc:1428] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:38:30.473747 25222 heartbeater.cc:503] Master 127.7.194.254:33351 was elected leader, sending a full tablet report...
I20260619 02:38:30.476258 25509 raft_consensus.cc:348] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } }
I20260619 02:38:30.476621 25509 raft_consensus.cc:374] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:30.476688 25509 raft_consensus.cc:729] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ec1043f0bf44e69fe14c8f338bbdde, State: Initialized, Role: FOLLOWER
I20260619 02:38:30.477245 25509 consensus_queue.cc:260] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } }
I20260619 02:38:30.477978 25356 heartbeater.cc:503] Master 127.7.194.254:33351 was elected leader, sending a full tablet report...
I20260619 02:38:30.478294 25509 ts_tablet_manager.cc:1428] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260619 02:38:30.592841 25513 raft_consensus.cc:482] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:30.593091 25513 raft_consensus.cc:504] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } }
I20260619 02:38:30.594491 25513 leader_election.cc:288] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33ec1043f0bf44e69fe14c8f338bbdde (127.7.194.194:44311), 8a5582aa846e494b8a2da3aedeaf31a1 (127.7.194.193:45045)
I20260619 02:38:30.598775 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.594746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:44311 (local address 127.7.194.195:47211)
0619 02:38:30.594949 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:30.594965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:30.595069 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:30.595437 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:30.595440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:30.595458 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:30.595575 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.595580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.596601 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.596607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.597476 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.597485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.598356 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.598360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.598469 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.598493 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:38:30.598578 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:38:30.598648 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 02:38:30.598848 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.595076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:47211 (local address 127.7.194.194:44311)
0619 02:38:30.595208 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.595212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.595227 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:38:30.595280 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.595283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.595329 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.595411 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:30.595723 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.596478 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.597643 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.598234 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.598332 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.598375 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.598625 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.598705 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:38:30.599146 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.594746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:45045 (local address 127.7.194.195:36049)
0619 02:38:30.595386 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:30.595402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:30.595557 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:30.595874 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:30.595877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:30.595897 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:30.596034 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.596041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.596848 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.596851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.597869 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:30.597879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.598627 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.598633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:30.598739 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.598764 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:38:30.598848 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 02:38:30.598926 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":271,"thread_start_us":152,"threads_started":1}
I20260619 02:38:30.599217 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.594865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:36049 (local address 127.7.194.193:45045)
0619 02:38:30.594997 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.595004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.595478 (+ 474us) server_negotiation.cc:407] Connection header received
0619 02:38:30.595718 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.595721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.595776 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.595835 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:30.596157 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.596735 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.597987 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.598499 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.598613 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.598668 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.599011 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.599103 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:38:30.599706 25310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66287d8a039c4585965928485b0c7913" candidate_uuid: "e03dd946636c462f9450ddac932fb4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" is_pre_election: true
I20260619 02:38:30.599925 25176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66287d8a039c4585965928485b0c7913" candidate_uuid: "e03dd946636c462f9450ddac932fb4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" is_pre_election: true
I20260619 02:38:30.600054 25310 raft_consensus.cc:2454] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e03dd946636c462f9450ddac932fb4a2 in term 0.
I20260619 02:38:30.600286 25176 raft_consensus.cc:2454] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e03dd946636c462f9450ddac932fb4a2 in term 0.
I20260619 02:38:30.600670 25380 leader_election.cc:302] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33ec1043f0bf44e69fe14c8f338bbdde, e03dd946636c462f9450ddac932fb4a2; no voters:
I20260619 02:38:30.601011 25513 raft_consensus.cc:2781] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:38:30.601099 25513 raft_consensus.cc:482] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:38:30.601164 25513 raft_consensus.cc:3037] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:30.602360 25513 raft_consensus.cc:504] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } }
I20260619 02:38:30.602785 25513 leader_election.cc:288] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [CANDIDATE]: Term 1 election: Requested vote from peers 33ec1043f0bf44e69fe14c8f338bbdde (127.7.194.194:44311), 8a5582aa846e494b8a2da3aedeaf31a1 (127.7.194.193:45045)
I20260619 02:38:30.603183 25310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66287d8a039c4585965928485b0c7913" candidate_uuid: "e03dd946636c462f9450ddac932fb4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33ec1043f0bf44e69fe14c8f338bbdde"
I20260619 02:38:30.603209 25176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66287d8a039c4585965928485b0c7913" candidate_uuid: "e03dd946636c462f9450ddac932fb4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5582aa846e494b8a2da3aedeaf31a1"
I20260619 02:38:30.603323 25176 raft_consensus.cc:3037] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:30.603323 25310 raft_consensus.cc:3037] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:30.604601 25310 raft_consensus.cc:2454] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e03dd946636c462f9450ddac932fb4a2 in term 1.
I20260619 02:38:30.604601 25176 raft_consensus.cc:2454] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e03dd946636c462f9450ddac932fb4a2 in term 1.
I20260619 02:38:30.604971 25379 leader_election.cc:302] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5582aa846e494b8a2da3aedeaf31a1, e03dd946636c462f9450ddac932fb4a2; no voters:
I20260619 02:38:30.605181 25513 raft_consensus.cc:2781] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:30.605386 25513 raft_consensus.cc:686] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 1 LEADER]: Becoming Leader. State: Replica: e03dd946636c462f9450ddac932fb4a2, State: Running, Role: LEADER
I20260619 02:38:30.605786 25513 consensus_queue.cc:237] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } }
I20260619 02:38:30.609107 25031 catalog_manager.cc:5495] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e03dd946636c462f9450ddac932fb4a2 (127.7.194.195). New cstate: current_term: 1 leader_uuid: "e03dd946636c462f9450ddac932fb4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e03dd946636c462f9450ddac932fb4a2" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42933 } health_report { overall_health: HEALTHY } } }
I20260619 02:38:30.640588 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.636946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.7.194.195:42933)
0619 02:38:30.637092 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.637096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.637111 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:38:30.637292 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.637302 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.637360 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.637469 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:38:30.637749 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.638262 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.639121 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.639360 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.639472 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.639782 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:38:30.640205 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:38:30.640339 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.640379 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.640435 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:38:30.646477 25176 raft_consensus.cc:1261] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [term 1 FOLLOWER]: Refusing update from remote peer e03dd946636c462f9450ddac932fb4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:30.646477 25310 raft_consensus.cc:1261] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [term 1 FOLLOWER]: Refusing update from remote peer e03dd946636c462f9450ddac932fb4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:30.647194 25513 consensus_queue.cc:1035] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5582aa846e494b8a2da3aedeaf31a1" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:30.647375 25517 consensus_queue.cc:1035] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ec1043f0bf44e69fe14c8f338bbdde" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:38:30.647879 25491 tablet.cc:2369] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:38:30.655166 25520 mvcc.cc:205] Tried to move back new op lower bound from 7298403166800781312 to 7298403166644330496. Current Snapshot: MvccSnapshot[applied={T|T < 7298403166800781312}]
I20260619 02:38:30.662210 25522 mvcc.cc:205] Tried to move back new op lower bound from 7298403166800781312 to 7298403166644330496. Current Snapshot: MvccSnapshot[applied={T|T < 7298403166800781312}]
I20260619 02:38:30.665774 25030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:30.665957 25030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:30.668994 25030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c4a17f4d07740dca368697b304e7342: Sending DeleteTablet for 3 replicas of tablet 66287d8a039c4585965928485b0c7913
I20260619 02:38:30.669847 25156 tablet_service.cc:1514] Processing DeleteTablet for tablet 66287d8a039c4585965928485b0c7913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:30 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:38650
I20260619 02:38:30.670012 25290 tablet_service.cc:1514] Processing DeleteTablet for tablet 66287d8a039c4585965928485b0c7913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:30 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:43298
I20260619 02:38:30.670635 25531 tablet_replica.cc:330] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1: stopping tablet replica
I20260619 02:38:30.670684 25424 tablet_service.cc:1514] Processing DeleteTablet for tablet 66287d8a039c4585965928485b0c7913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:30 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:53670
I20260619 02:38:30.671012 25531 raft_consensus.cc:2229] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:30.671241 25533 tablet_replica.cc:330] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: stopping tablet replica
I20260619 02:38:30.671429 25531 raft_consensus.cc:2258] T 66287d8a039c4585965928485b0c7913 P 8a5582aa846e494b8a2da3aedeaf31a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:30.671622 25533 raft_consensus.cc:2229] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:38:30.671643 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 25093
I20260619 02:38:30.672375 25533 raft_consensus.cc:2258] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:30.673255 25532 tablet_replica.cc:330] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: stopping tablet replica
I20260619 02:38:30.673650 25532 raft_consensus.cc:2229] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:30.674016 25532 raft_consensus.cc:2258] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:30.674463 25533 ts_tablet_manager.cc:1905] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:30.676256 25532 ts_tablet_manager.cc:1905] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:30.677821 25533 ts_tablet_manager.cc:1918] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:30.678004 25533 log.cc:1196] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/66287d8a039c4585965928485b0c7913
I20260619 02:38:30.678458 25533 ts_tablet_manager.cc:1939] T 66287d8a039c4585965928485b0c7913 P e03dd946636c462f9450ddac932fb4a2: Deleting consensus metadata
I20260619 02:38:30.679355 25532 ts_tablet_manager.cc:1918] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:30.679526 25532 log.cc:1196] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/66287d8a039c4585965928485b0c7913
I20260619 02:38:30.679924 25532 ts_tablet_manager.cc:1939] T 66287d8a039c4585965928485b0c7913 P 33ec1043f0bf44e69fe14c8f338bbdde: Deleting consensus metadata
I20260619 02:38:30.680028 25017 catalog_manager.cc:4841] TS e03dd946636c462f9450ddac932fb4a2 (127.7.194.195:42933): tablet 66287d8a039c4585965928485b0c7913 (table test-table [id=f7baf0acdefa4ddc932a17d79074a1da]) successfully deleted
I20260619 02:38:30.680363 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 25227
W20260619 02:38:30.680342 25017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:45045 (error 108)
I20260619 02:38:30.682662 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.682364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:45045 (local address 127.0.0.1:38654)
0619 02:38:30.682501 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:30.682579 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:38:30.682893 25017 catalog_manager.cc:4584] TS 8a5582aa846e494b8a2da3aedeaf31a1 (127.7.194.193:45045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66287d8a039c4585965928485b0c7913: Network error: Client connection negotiation failed: client connection to 127.7.194.193:45045: connect: Connection refused (error 111)
I20260619 02:38:30.686753 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 25361
I20260619 02:38:30.688400 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.687028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:44311 (local address 127.0.0.1:43312)
0619 02:38:30.688287 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:30.688334 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.194:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1185}
I20260619 02:38:30.692545 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 24999
2026-06-19T02:38:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2390 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T02:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:30Z Disabled control of system clock
I20260619 02:38:30.722656 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44615
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:38:30.831487 25538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:30.831792 25538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:30.831831 25538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:30.835816 25538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:30.835883 25538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:30.835898 25538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:30.835915 25538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:30.835932 25538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:30.840822 25538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:44615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:30.841979 25538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:30.843349 25538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:30.851969 25543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:30.851934 25546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:30.851918 25544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:30.852348 25538 server_base.cc:1034] running on GCE node
I20260619 02:38:30.852957 25538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:30.854069 25538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:30.855252 25538 hybrid_clock.cc:648] HybridClock initialized: now 1781836710855237 us; error 28 us; skew 500 ppm
I20260619 02:38:30.857363 25538 webserver.cc:460] Webserver started at http://127.7.194.254:39155/ using document root <none> and password file <none>
I20260619 02:38:30.858003 25538 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:30.858068 25538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:30.858325 25538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:30.860431 25538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "4095817c37f54afaa94576c56c09e9c0"
format_stamp: "Formatted at 2026-06-19 02:38:30 on dist-test-slave-9tkz"
server_key: "846dfa2bbdcdb880b3ebdb1286dd0d28"
server_key_iv: "db10c2884f5be0830bf30d9891f0e521"
server_key_version: "encryptionkey@0"
I20260619 02:38:30.861025 25538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "4095817c37f54afaa94576c56c09e9c0"
format_stamp: "Formatted at 2026-06-19 02:38:30 on dist-test-slave-9tkz"
server_key: "846dfa2bbdcdb880b3ebdb1286dd0d28"
server_key_iv: "db10c2884f5be0830bf30d9891f0e521"
server_key_version: "encryptionkey@0"
I20260619 02:38:30.864807 25538 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:38:30.867437 25552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:30.868657 25538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:38:30.868798 25538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "4095817c37f54afaa94576c56c09e9c0"
format_stamp: "Formatted at 2026-06-19 02:38:30 on dist-test-slave-9tkz"
server_key: "846dfa2bbdcdb880b3ebdb1286dd0d28"
server_key_iv: "db10c2884f5be0830bf30d9891f0e521"
server_key_version: "encryptionkey@0"
I20260619 02:38:30.868909 25538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:30.883002 25538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:30.883744 25538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:30.883942 25538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:30.891296 25538 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:44615
I20260619 02:38:30.891310 25604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:44615 every 8 connection(s)
I20260619 02:38:30.892306 25538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:30.895501 25605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:30.898111 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 25538
I20260619 02:38:30.898262 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:30.898566 7947 external_mini_cluster.cc:1428] Setting key ae47d00197e792aa99c1f138acf72702
I20260619 02:38:30.901449 25605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0: Bootstrap starting.
I20260619 02:38:30.903738 25605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:30.904475 25605 log.cc:824] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:30.906411 25605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0: No bootstrap required, opened a new log
I20260619 02:38:30.907919 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:30.899946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.7.194.254:44615)
0619 02:38:30.900628 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:30.900652 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:30.900699 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:38:30.901347 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:30.901364 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:30.901828 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:30.902168 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:30.902617 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.903572 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.904537 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:30.905062 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:30.905262 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:30.906192 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:30.906203 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:30.906216 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:30.906651 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:30.906711 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:30.906731 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:30.906835 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:30.906973 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 02:38:30.907232 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
I20260619 02:38:30.909803 25605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095817c37f54afaa94576c56c09e9c0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44615 } }
I20260619 02:38:30.910068 25605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:30.910131 25605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4095817c37f54afaa94576c56c09e9c0, State: Initialized, Role: FOLLOWER
I20260619 02:38:30.910581 25605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095817c37f54afaa94576c56c09e9c0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44615 } }
I20260619 02:38:30.910749 25605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:30.910799 25605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:30.910868 25605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:30.911742 25605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095817c37f54afaa94576c56c09e9c0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44615 } }
I20260619 02:38:30.912055 25605 leader_election.cc:302] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4095817c37f54afaa94576c56c09e9c0; no voters:
I20260619 02:38:30.912315 25605 leader_election.cc:288] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:30.912437 25610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:30.912724 25610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4095817c37f54afaa94576c56c09e9c0, State: Running, Role: LEADER
I20260619 02:38:30.913089 25610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095817c37f54afaa94576c56c09e9c0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44615 } }
I20260619 02:38:30.913734 25605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:30.914956 25612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4095817c37f54afaa94576c56c09e9c0. Latest consensus state: current_term: 1 leader_uuid: "4095817c37f54afaa94576c56c09e9c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095817c37f54afaa94576c56c09e9c0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44615 } } }
I20260619 02:38:30.914958 25611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4095817c37f54afaa94576c56c09e9c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095817c37f54afaa94576c56c09e9c0" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44615 } } }
I20260619 02:38:30.915102 25612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:30.915107 25611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:30.915488 25619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:30.918500 25619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:30.924139 25619 catalog_manager.cc:1347] Generated new cluster ID: feb703fb5f2b4ed9a56782f302783efe
I20260619 02:38:30.924240 25619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:30.942193 25619 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:30.943234 25619 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:30.949501 25619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0: Generated new TSK 0
I20260619 02:38:30.950244 25619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:30.958725 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44615
--builtin_ntp_servers=127.7.194.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:38:31.081709 25629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:31.081969 25629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:31.082008 25629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:31.086084 25629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:31.086139 25629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:31.086229 25629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:38:31.090950 25629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:31.092064 25629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:31.093468 25629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:31.103137 25635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:31.103129 25637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:31.103148 25634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:31.103374 25629 server_base.cc:1034] running on GCE node
I20260619 02:38:31.103796 25629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:31.104485 25629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:31.105620 25629 hybrid_clock.cc:648] HybridClock initialized: now 1781836711105602 us; error 28 us; skew 500 ppm
I20260619 02:38:31.107502 25629 webserver.cc:460] Webserver started at http://127.7.194.193:44923/ using document root <none> and password file <none>
I20260619 02:38:31.108090 25629 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:31.108186 25629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:31.108412 25629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:31.110527 25629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "e2faaa62b6f245bdb110ef7cafc2b67c"
format_stamp: "Formatted at 2026-06-19 02:38:31 on dist-test-slave-9tkz"
server_key: "4d08823d5e0e54b1000bb22436831051"
server_key_iv: "dfde0e28508f14e7540671fe07012233"
server_key_version: "encryptionkey@0"
I20260619 02:38:31.111132 25629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "e2faaa62b6f245bdb110ef7cafc2b67c"
format_stamp: "Formatted at 2026-06-19 02:38:31 on dist-test-slave-9tkz"
server_key: "4d08823d5e0e54b1000bb22436831051"
server_key_iv: "dfde0e28508f14e7540671fe07012233"
server_key_version: "encryptionkey@0"
I20260619 02:38:31.115013 25629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:38:31.117409 25643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:31.118669 25629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:31.118814 25629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "e2faaa62b6f245bdb110ef7cafc2b67c"
format_stamp: "Formatted at 2026-06-19 02:38:31 on dist-test-slave-9tkz"
server_key: "4d08823d5e0e54b1000bb22436831051"
server_key_iv: "dfde0e28508f14e7540671fe07012233"
server_key_version: "encryptionkey@0"
I20260619 02:38:31.118927 25629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:31.142385 25629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:31.143133 25629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:31.143352 25629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:31.143966 25629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:31.144961 25629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:31.145033 25629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:31.145102 25629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:31.145154 25629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:31.153641 25629 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:38017
I20260619 02:38:31.153653 25756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:38017 every 8 connection(s)
I20260619 02:38:31.154662 25629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:38:31.156005 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 25629
I20260619 02:38:31.156117 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:38:31.156520 7947 external_mini_cluster.cc:1428] Setting key 6722a81774247e9b2a21980e1ca93a7b
I20260619 02:38:31.158799 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44615
--builtin_ntp_servers=127.7.194.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:38:31.165340 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.156619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:53867 (local address 127.7.194.254:44615)
0619 02:38:31.156761 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.156766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.158068 (+ 1302us) server_negotiation.cc:407] Connection header received
0619 02:38:31.160033 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.160035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.160078 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.160150 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:31.161146 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.161891 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.163022 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.163427 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.163560 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.164705 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:31.164712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:31.164716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:31.164734 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:31.164760 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:31.164774 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:31.164865 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.165044 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.165127 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:38:31.166254 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.157083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44615 (local address 127.7.194.193:53867)
0619 02:38:31.157932 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:31.157990 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:31.159793 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:31.160347 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:31.160360 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:31.160485 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:31.161008 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.161022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.162003 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.162005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.162899 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.162908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.163587 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.163590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.163743 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.164422 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:38:31.164424 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:31.164466 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:38:31.164531 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:31.164888 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:31.164896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:31.164902 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:38:31.165012 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:38:31.165273 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":151,"threads_started":1}
I20260619 02:38:31.167546 25757 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44615
I20260619 02:38:31.167778 25757 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:31.168309 25757 heartbeater.cc:511] Master 127.7.194.254:44615 requested a full tablet report, sending...
I20260619 02:38:31.169972 25569 ts_manager.cc:194] Registered new tserver with Master: e2faaa62b6f245bdb110ef7cafc2b67c (127.7.194.193:38017)
I20260619 02:38:31.171537 25569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.193:53867
W20260619 02:38:31.272015 25760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:31.272375 25760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:31.272465 25760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:31.276618 25760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:31.276711 25760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:31.276803 25760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:38:31.281620 25760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:31.282820 25760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:31.284330 25760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:31.294134 25765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:31.294191 25766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:31.294134 25768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:31.294432 25760 server_base.cc:1034] running on GCE node
I20260619 02:38:31.294898 25760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:31.295548 25760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:31.296718 25760 hybrid_clock.cc:648] HybridClock initialized: now 1781836711296704 us; error 51 us; skew 500 ppm
I20260619 02:38:31.298712 25760 webserver.cc:460] Webserver started at http://127.7.194.194:34151/ using document root <none> and password file <none>
I20260619 02:38:31.299345 25760 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:31.299408 25760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:31.299669 25760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:31.301890 25760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "a4ee420241e54b33b96b25c3c7d12832"
format_stamp: "Formatted at 2026-06-19 02:38:31 on dist-test-slave-9tkz"
server_key: "290a3ec5d58beec5954220a4e268b23a"
server_key_iv: "543b4eae64cc6e1e3a3f9520c6989b1d"
server_key_version: "encryptionkey@0"
I20260619 02:38:31.302470 25760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "a4ee420241e54b33b96b25c3c7d12832"
format_stamp: "Formatted at 2026-06-19 02:38:31 on dist-test-slave-9tkz"
server_key: "290a3ec5d58beec5954220a4e268b23a"
server_key_iv: "543b4eae64cc6e1e3a3f9520c6989b1d"
server_key_version: "encryptionkey@0"
I20260619 02:38:31.306365 25760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:38:31.308760 25774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:31.310096 25760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:31.310225 25760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "a4ee420241e54b33b96b25c3c7d12832"
format_stamp: "Formatted at 2026-06-19 02:38:31 on dist-test-slave-9tkz"
server_key: "290a3ec5d58beec5954220a4e268b23a"
server_key_iv: "543b4eae64cc6e1e3a3f9520c6989b1d"
server_key_version: "encryptionkey@0"
I20260619 02:38:31.310335 25760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:31.340600 25760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:31.341380 25760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:31.341619 25760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:31.342262 25760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:31.343250 25760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:31.343322 25760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:31.343400 25760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:31.343441 25760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:31.353233 25760 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:45477
I20260619 02:38:31.353322 25887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:45477 every 8 connection(s)
I20260619 02:38:31.354282 25760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:38:31.356477 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 25760
I20260619 02:38:31.356608 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:38:31.356891 7947 external_mini_cluster.cc:1428] Setting key 032014efffa1c4efbf680a8ec8429810
I20260619 02:38:31.358971 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44615
--builtin_ntp_servers=127.7.194.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:38:31.364799 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.356167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:44571 (local address 127.7.194.254:44615)
0619 02:38:31.356340 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.356345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.357588 (+ 1243us) server_negotiation.cc:407] Connection header received
0619 02:38:31.359134 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.359137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.359189 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.359267 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:31.360224 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.360767 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.362578 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.362869 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.362954 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.364291 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:31.364294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:31.364296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:31.364308 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:31.364327 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:31.364334 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:31.364392 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.364621 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.364674 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:38:31.366153 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.356632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44615 (local address 127.7.194.194:44571)
0619 02:38:31.357407 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:31.357459 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:31.358920 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:31.359435 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:31.359442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:31.359528 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:31.360022 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.360064 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.360906 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.360912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.362455 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.362469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.363039 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.363045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.363294 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.363971 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:38:31.363976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:31.364037 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:38:31.364132 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:31.364423 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:31.364435 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:31.364443 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:38:31.364631 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 02:38:31.364969 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260619 02:38:31.367321 25888 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44615
I20260619 02:38:31.367609 25888 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:31.368290 25888 heartbeater.cc:511] Master 127.7.194.254:44615 requested a full tablet report, sending...
I20260619 02:38:31.369416 25569 ts_manager.cc:194] Registered new tserver with Master: a4ee420241e54b33b96b25c3c7d12832 (127.7.194.194:45477)
I20260619 02:38:31.370030 25569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.194:44571
W20260619 02:38:31.472733 25891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:31.473097 25891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:31.473136 25891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:31.477219 25891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:31.477272 25891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:31.477351 25891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:38:31.482107 25891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:31.483179 25891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:31.484582 25891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:31.493983 25896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:31.494076 25891 server_base.cc:1034] running on GCE node
W20260619 02:38:31.494024 25899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:31.493996 25897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:31.494637 25891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:31.495273 25891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:31.496443 25891 hybrid_clock.cc:648] HybridClock initialized: now 1781836711496422 us; error 33 us; skew 500 ppm
I20260619 02:38:31.498286 25891 webserver.cc:460] Webserver started at http://127.7.194.195:43667/ using document root <none> and password file <none>
I20260619 02:38:31.498850 25891 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:31.498935 25891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:31.499176 25891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:31.501320 25891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "29cebc2b2ce042af8b4c9ea7c13adf90"
format_stamp: "Formatted at 2026-06-19 02:38:31 on dist-test-slave-9tkz"
server_key: "f65f657670db91e2ac93eaa0e58a3cf5"
server_key_iv: "90be3a982b817d50c73f515162966895"
server_key_version: "encryptionkey@0"
I20260619 02:38:31.501937 25891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "29cebc2b2ce042af8b4c9ea7c13adf90"
format_stamp: "Formatted at 2026-06-19 02:38:31 on dist-test-slave-9tkz"
server_key: "f65f657670db91e2ac93eaa0e58a3cf5"
server_key_iv: "90be3a982b817d50c73f515162966895"
server_key_version: "encryptionkey@0"
I20260619 02:38:31.505749 25891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:38:31.508293 25905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:31.509493 25891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:31.509670 25891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "29cebc2b2ce042af8b4c9ea7c13adf90"
format_stamp: "Formatted at 2026-06-19 02:38:31 on dist-test-slave-9tkz"
server_key: "f65f657670db91e2ac93eaa0e58a3cf5"
server_key_iv: "90be3a982b817d50c73f515162966895"
server_key_version: "encryptionkey@0"
I20260619 02:38:31.509774 25891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:31.572289 25891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:31.573040 25891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:31.573293 25891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:31.573993 25891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:31.574970 25891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:31.575016 25891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:31.575085 25891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:31.575132 25891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:31.584951 25891 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:39105
I20260619 02:38:31.584980 26018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:39105 every 8 connection(s)
I20260619 02:38:31.585981 25891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:38:31.586192 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 25891
I20260619 02:38:31.586284 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:38:31.586634 7947 external_mini_cluster.cc:1428] Setting key dc754f5c5af1bbc886b9c08acfa016df
I20260619 02:38:31.595902 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.588141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:54673 (local address 127.7.194.254:44615)
0619 02:38:31.588271 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.588276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.589437 (+ 1161us) server_negotiation.cc:407] Connection header received
0619 02:38:31.590872 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.590876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.590929 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.591008 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:31.591983 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.592559 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.593866 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.594112 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.594227 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.595389 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:31.595393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:31.595396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:31.595409 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:31.595438 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:31.595446 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:31.595521 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.595692 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.595758 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:38:31.596769 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.588596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44615 (local address 127.7.194.195:54673)
0619 02:38:31.589301 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:31.589341 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:31.590670 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:31.591204 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:31.591212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:31.591302 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:31.591846 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.591858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.592691 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.592695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.593747 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.593757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.594272 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.594276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.594457 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.595110 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:38:31.595112 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:31.595152 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:38:31.595225 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:31.595534 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:31.595543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:31.595549 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:38:31.595685 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 02:38:31.595979 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":151,"threads_started":1}
I20260619 02:38:31.597950 26019 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44615
I20260619 02:38:31.598218 26019 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:31.598753 26019 heartbeater.cc:511] Master 127.7.194.254:44615 requested a full tablet report, sending...
I20260619 02:38:31.599888 25568 ts_manager.cc:194] Registered new tserver with Master: 29cebc2b2ce042af8b4c9ea7c13adf90 (127.7.194.195:39105)
I20260619 02:38:31.600503 25568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.195:54673
I20260619 02:38:31.600713 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:38:31.606793 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.602810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.7.194.254:44615)
0619 02:38:31.602917 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.602922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.603155 (+ 233us) server_negotiation.cc:407] Connection header received
0619 02:38:31.603247 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.603249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.603288 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.603346 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:31.603589 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.604066 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.604888 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.605113 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.605230 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.606352 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:31.606355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:31.606357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:31.606368 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:31.606385 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:31.606393 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:31.606453 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.606562 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.606612 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:38:31.609815 25568 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:38:31.612854 25568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:38:31.627738 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.621906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.7.194.194:45477)
0619 02:38:31.622907 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.622920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.622946 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:38:31.623027 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.623034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.623282 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.623451 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:31.623746 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.624699 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.626324 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.627042 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.627186 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.627386 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.627462 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.627559 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":122,"threads_started":1}
I20260619 02:38:31.627727 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.621791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:45477 (local address 127.0.0.1:43230)
0619 02:38:31.622158 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:31.622206 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:31.622325 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:31.623452 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:31.623458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:31.623482 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:31.623613 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.623619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.624807 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.624813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.626213 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.626224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.627198 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.627202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.627312 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.627334 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:38:31.627433 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:38:31.627555 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":25,"thread_start_us":91,"threads_started":1}
I20260619 02:38:31.627736 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.622107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38017 (local address 127.0.0.1:41360)
0619 02:38:31.622562 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:31.622595 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:31.622683 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:31.623781 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:31.623784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:31.623801 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:31.623880 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.623886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.625116 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.625120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.626252 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.626263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.627139 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.627143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.627262 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.627323 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 02:38:31.627409 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:38:31.627563 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":54,"thread_start_us":111,"threads_started":1}
I20260619 02:38:31.628583 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.622572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.7.194.193:38017)
0619 02:38:31.623338 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.623346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.623372 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:38:31.623451 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.623456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.623630 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.623807 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:31.623966 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.625001 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.626387 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.627005 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.628081 (+ 1076us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.628269 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.628320 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.628428 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":112,"threads_started":1}
I20260619 02:38:31.628575 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.622499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:39105 (local address 127.0.0.1:43852)
0619 02:38:31.622996 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:31.623009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:31.623098 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:31.624075 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:31.624078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:31.624095 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:31.624167 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.624173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.625472 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.625477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.626723 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.626734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.628218 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.628222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.628308 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.628331 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:38:31.628391 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:38:31.628449 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":102,"threads_started":1}
I20260619 02:38:31.629609 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.623065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.7.194.195:39105)
0619 02:38:31.623415 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.623425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.623447 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:38:31.623523 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.623528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.623800 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.623938 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:31.624292 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.625345 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.626865 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.627626 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.627738 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.629316 (+ 1578us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.629358 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.629432 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":140,"threads_started":1}
I20260619 02:38:31.631129 25811 tablet_service.cc:1467] Processing CreateTablet for tablet 938634a0ccff4757b333aeb64dcf3174 (DEFAULT_TABLE table=test-table [id=b10eb94a133a4a9aa6f9e4714893a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:31.631345 25691 tablet_service.cc:1467] Processing CreateTablet for tablet 938634a0ccff4757b333aeb64dcf3174 (DEFAULT_TABLE table=test-table [id=b10eb94a133a4a9aa6f9e4714893a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:31.631815 25953 tablet_service.cc:1467] Processing CreateTablet for tablet 938634a0ccff4757b333aeb64dcf3174 (DEFAULT_TABLE table=test-table [id=b10eb94a133a4a9aa6f9e4714893a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:31.632278 25811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938634a0ccff4757b333aeb64dcf3174. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:31.632278 25691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938634a0ccff4757b333aeb64dcf3174. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:31.632767 25953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938634a0ccff4757b333aeb64dcf3174. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:31.638655 26036 tablet_bootstrap.cc:492] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: Bootstrap starting.
I20260619 02:38:31.640177 26037 tablet_bootstrap.cc:492] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: Bootstrap starting.
I20260619 02:38:31.640298 26038 tablet_bootstrap.cc:492] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c: Bootstrap starting.
I20260619 02:38:31.640890 26036 tablet_bootstrap.cc:654] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:31.641826 26036 log.cc:824] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:31.642350 26038 tablet_bootstrap.cc:654] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:31.642351 26037 tablet_bootstrap.cc:654] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:31.643246 26038 log.cc:824] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:31.643246 26037 log.cc:824] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:31.643778 26036 tablet_bootstrap.cc:492] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: No bootstrap required, opened a new log
I20260619 02:38:31.643985 26036 ts_tablet_manager.cc:1397] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:38:31.645236 26038 tablet_bootstrap.cc:492] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c: No bootstrap required, opened a new log
I20260619 02:38:31.645427 26038 ts_tablet_manager.cc:1397] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:38:31.645481 26037 tablet_bootstrap.cc:492] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: No bootstrap required, opened a new log
I20260619 02:38:31.645725 26037 ts_tablet_manager.cc:1397] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:38:31.647500 26036 raft_consensus.cc:348] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } }
I20260619 02:38:31.647804 26036 raft_consensus.cc:374] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:31.647876 26036 raft_consensus.cc:729] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cebc2b2ce042af8b4c9ea7c13adf90, State: Initialized, Role: FOLLOWER
I20260619 02:38:31.648345 26036 consensus_queue.cc:260] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } }
I20260619 02:38:31.648998 26019 heartbeater.cc:503] Master 127.7.194.254:44615 was elected leader, sending a full tablet report...
I20260619 02:38:31.649371 26036 ts_tablet_manager.cc:1428] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:38:31.649869 26038 raft_consensus.cc:348] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } }
I20260619 02:38:31.650134 26038 raft_consensus.cc:374] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:31.650027 26037 raft_consensus.cc:348] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } }
I20260619 02:38:31.650245 26038 raft_consensus.cc:729] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2faaa62b6f245bdb110ef7cafc2b67c, State: Initialized, Role: FOLLOWER
I20260619 02:38:31.650277 26037 raft_consensus.cc:374] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:31.650328 26037 raft_consensus.cc:729] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ee420241e54b33b96b25c3c7d12832, State: Initialized, Role: FOLLOWER
I20260619 02:38:31.650700 26038 consensus_queue.cc:260] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } }
I20260619 02:38:31.650700 26037 consensus_queue.cc:260] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } }
I20260619 02:38:31.651572 25888 heartbeater.cc:503] Master 127.7.194.254:44615 was elected leader, sending a full tablet report...
I20260619 02:38:31.651662 26038 ts_tablet_manager.cc:1428] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:38:31.651678 26037 ts_tablet_manager.cc:1428] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:38:31.652635 25757 heartbeater.cc:503] Master 127.7.194.254:44615 was elected leader, sending a full tablet report...
W20260619 02:38:31.655827 25758 tablet.cc:2369] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:38:31.837090 26020 tablet.cc:2369] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:38:31.849464 26044 raft_consensus.cc:482] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:31.849754 26044 raft_consensus.cc:504] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } }
I20260619 02:38:31.851013 26044 leader_election.cc:288] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4ee420241e54b33b96b25c3c7d12832 (127.7.194.194:45477), 29cebc2b2ce042af8b4c9ea7c13adf90 (127.7.194.195:39105)
W20260619 02:38:31.855377 25889 tablet.cc:2369] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:38:31.856086 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.851273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:45477 (local address 127.7.194.193:33659)
0619 02:38:31.851588 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:31.851605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:31.851723 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:31.852099 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:31.852105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:31.852127 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:31.852278 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.852283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.853303 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.853305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.854678 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.854699 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.855644 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.855648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.855745 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.855770 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:38:31.855848 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:38:31.855918 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":65,"thread_start_us":104,"threads_started":1}
I20260619 02:38:31.856104 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.851376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:33659 (local address 127.7.194.194:45477)
0619 02:38:31.851520 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.851523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.851691 (+ 168us) server_negotiation.cc:407] Connection header received
0619 02:38:31.851903 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.851906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.851965 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.852069 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:31.852381 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.853187 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.854827 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.855413 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.855542 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.855603 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.855883 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.855985 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:38:31.856104 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.851876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:50071 (local address 127.7.194.195:39105)
0619 02:38:31.852005 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.852012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.852178 (+ 166us) server_negotiation.cc:407] Connection header received
0619 02:38:31.852249 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.852251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.852288 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.852378 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:31.852628 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.853399 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.854828 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.855456 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.855552 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.855603 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.855932 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.856001 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:38:31.856531 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.851706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:39105 (local address 127.7.194.193:50071)
0619 02:38:31.852028 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:31.852063 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:31.852146 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:31.852399 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:31.852401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:31.852417 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:31.852517 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.852522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.853486 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.853489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.854692 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:31.854699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.855696 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.855699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:31.855774 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.855792 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:38:31.856307 (+ 515us) client_negotiation.cc:240] Negotiation successful
0619 02:38:31.856401 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":4,"thread_start_us":110,"threads_started":1}
I20260619 02:38:31.856873 25842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938634a0ccff4757b333aeb64dcf3174" candidate_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ee420241e54b33b96b25c3c7d12832" is_pre_election: true
I20260619 02:38:31.857237 25842 raft_consensus.cc:2454] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2faaa62b6f245bdb110ef7cafc2b67c in term 0.
I20260619 02:38:31.857252 25973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938634a0ccff4757b333aeb64dcf3174" candidate_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" is_pre_election: true
I20260619 02:38:31.857594 25973 raft_consensus.cc:2454] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2faaa62b6f245bdb110ef7cafc2b67c in term 0.
I20260619 02:38:31.857751 25645 leader_election.cc:302] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ee420241e54b33b96b25c3c7d12832, e2faaa62b6f245bdb110ef7cafc2b67c; no voters:
I20260619 02:38:31.858062 26044 raft_consensus.cc:2781] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:38:31.858171 26044 raft_consensus.cc:482] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:38:31.858227 26044 raft_consensus.cc:3037] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:31.859349 26044 raft_consensus.cc:504] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } }
I20260619 02:38:31.859679 26044 leader_election.cc:288] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [CANDIDATE]: Term 1 election: Requested vote from peers a4ee420241e54b33b96b25c3c7d12832 (127.7.194.194:45477), 29cebc2b2ce042af8b4c9ea7c13adf90 (127.7.194.195:39105)
I20260619 02:38:31.860072 25973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938634a0ccff4757b333aeb64dcf3174" candidate_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90"
I20260619 02:38:31.860069 25842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938634a0ccff4757b333aeb64dcf3174" candidate_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ee420241e54b33b96b25c3c7d12832"
I20260619 02:38:31.860199 25973 raft_consensus.cc:3037] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:31.860208 25842 raft_consensus.cc:3037] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:31.861510 25842 raft_consensus.cc:2454] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2faaa62b6f245bdb110ef7cafc2b67c in term 1.
I20260619 02:38:31.861510 25973 raft_consensus.cc:2454] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2faaa62b6f245bdb110ef7cafc2b67c in term 1.
I20260619 02:38:31.861840 25645 leader_election.cc:302] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ee420241e54b33b96b25c3c7d12832, e2faaa62b6f245bdb110ef7cafc2b67c; no voters:
I20260619 02:38:31.861999 26044 raft_consensus.cc:2781] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:31.862231 26044 raft_consensus.cc:686] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 1 LEADER]: Becoming Leader. State: Replica: e2faaa62b6f245bdb110ef7cafc2b67c, State: Running, Role: LEADER
I20260619 02:38:31.862571 26044 consensus_queue.cc:237] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } }
I20260619 02:38:31.865702 25569 catalog_manager.cc:5495] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c reported cstate change: term changed from 0 to 1, leader changed from <none> to e2faaa62b6f245bdb110ef7cafc2b67c (127.7.194.193). New cstate: current_term: 1 leader_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2faaa62b6f245bdb110ef7cafc2b67c" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 38017 } health_report { overall_health: HEALTHY } } }
I20260619 02:38:31.923684 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.920136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.7.194.193:38017)
0619 02:38:31.920265 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:31.920268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:31.920301 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:38:31.920380 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:31.920382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:31.920439 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:31.920518 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:38:31.920726 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.921248 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.922160 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:31.922441 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:31.922566 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:31.922809 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:38:31.923273 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:38:31.923395 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:31.923460 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 02:38:31.923515 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:38:31.929569 25842 raft_consensus.cc:1261] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [term 1 FOLLOWER]: Refusing update from remote peer e2faaa62b6f245bdb110ef7cafc2b67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:31.929600 25973 raft_consensus.cc:1261] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [term 1 FOLLOWER]: Refusing update from remote peer e2faaa62b6f245bdb110ef7cafc2b67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:31.930296 26044 consensus_queue.cc:1035] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ee420241e54b33b96b25c3c7d12832" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:31.930506 26047 consensus_queue.cc:1035] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cebc2b2ce042af8b4c9ea7c13adf90" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:31.940135 26051 mvcc.cc:205] Tried to move back new op lower bound from 7298403172056080384 to 7298403171791732736. Current Snapshot: MvccSnapshot[applied={T|T < 7298403172056080384}]
I20260619 02:38:31.946135 25567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:31.946328 25567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:31.948800 25567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4095817c37f54afaa94576c56c09e9c0: Sending DeleteTablet for 3 replicas of tablet 938634a0ccff4757b333aeb64dcf3174
I20260619 02:38:31.949671 25691 tablet_service.cc:1514] Processing DeleteTablet for tablet 938634a0ccff4757b333aeb64dcf3174 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:31 UTC) from {username='slave'} at 127.0.0.1:41360
I20260619 02:38:31.949695 25953 tablet_service.cc:1514] Processing DeleteTablet for tablet 938634a0ccff4757b333aeb64dcf3174 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:31 UTC) from {username='slave'} at 127.0.0.1:43852
I20260619 02:38:31.949841 25811 tablet_service.cc:1514] Processing DeleteTablet for tablet 938634a0ccff4757b333aeb64dcf3174 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:31 UTC) from {username='slave'} at 127.0.0.1:43230
I20260619 02:38:31.950264 26061 tablet_replica.cc:330] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: stopping tablet replica
I20260619 02:38:31.950377 26060 tablet_replica.cc:330] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c: stopping tablet replica
I20260619 02:38:31.950591 26061 raft_consensus.cc:2229] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:31.950647 26060 raft_consensus.cc:2229] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:38:31.950837 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 25629
I20260619 02:38:31.950922 26061 raft_consensus.cc:2258] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:31.950984 26060 raft_consensus.cc:2258] T 938634a0ccff4757b333aeb64dcf3174 P e2faaa62b6f245bdb110ef7cafc2b67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:31.951632 26059 tablet_replica.cc:330] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: stopping tablet replica
I20260619 02:38:31.951932 26059 raft_consensus.cc:2229] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:31.952219 26059 raft_consensus.cc:2258] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:31.953749 26061 ts_tablet_manager.cc:1905] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:31.954317 26059 ts_tablet_manager.cc:1905] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:31.956661 26061 ts_tablet_manager.cc:1918] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:31.956768 26061 log.cc:1196] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/938634a0ccff4757b333aeb64dcf3174
I20260619 02:38:31.957007 26059 ts_tablet_manager.cc:1918] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:31.957110 26059 log.cc:1196] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/938634a0ccff4757b333aeb64dcf3174
I20260619 02:38:31.957116 26061 ts_tablet_manager.cc:1939] T 938634a0ccff4757b333aeb64dcf3174 P a4ee420241e54b33b96b25c3c7d12832: Deleting consensus metadata
I20260619 02:38:31.957412 26059 ts_tablet_manager.cc:1939] T 938634a0ccff4757b333aeb64dcf3174 P 29cebc2b2ce042af8b4c9ea7c13adf90: Deleting consensus metadata
W20260619 02:38:31.957767 25555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:38017 (error 108)
I20260619 02:38:31.958500 25554 catalog_manager.cc:4841] TS 29cebc2b2ce042af8b4c9ea7c13adf90 (127.7.194.195:39105): tablet 938634a0ccff4757b333aeb64dcf3174 (table test-table [id=b10eb94a133a4a9aa6f9e4714893a52c]) successfully deleted
I20260619 02:38:31.958719 25554 catalog_manager.cc:4841] TS a4ee420241e54b33b96b25c3c7d12832 (127.7.194.194:45477): tablet 938634a0ccff4757b333aeb64dcf3174 (table test-table [id=b10eb94a133a4a9aa6f9e4714893a52c]) successfully deleted
I20260619 02:38:31.958827 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:31.958500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:38017 (local address 127.0.0.1:41386)
0619 02:38:31.958631 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:31.958716 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:38:31.958912 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 25760
W20260619 02:38:31.960407 25555 catalog_manager.cc:4584] TS e2faaa62b6f245bdb110ef7cafc2b67c (127.7.194.193:38017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 938634a0ccff4757b333aeb64dcf3174: Network error: Client connection negotiation failed: client connection to 127.7.194.193:38017: connect: Connection refused (error 111)
I20260619 02:38:31.965413 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 25891
I20260619 02:38:31.971516 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 25538
2026-06-19T02:38:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1278 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:32 dist-test-slave-9tkz krb5kdc[26065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:32 dist-test-slave-9tkz krb5kdc[26065](info): set up 2 sockets
Jun 19 02:38:32 dist-test-slave-9tkz krb5kdc[26065](info): commencing operation
krb5kdc: starting...
W20260619 02:38:33.013480 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:33 dist-test-slave-9tkz krb5kdc[26065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:38:33.136154 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38915
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:33.250103 26078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:33.250448 26078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:33.250536 26078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:33.254698 26078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:33.254797 26078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:33.254837 26078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:33.254900 26078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:33.254936 26078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:33.260239 26078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:33.261417 26078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:33.262517 26078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:33.271206 26086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:33.271287 26078 server_base.cc:1034] running on GCE node
W20260619 02:38:33.271237 26083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:33.271153 26084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:33.272076 26078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:33.273216 26078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:33.274417 26078 hybrid_clock.cc:648] HybridClock initialized: now 1781836713274397 us; error 29 us; skew 500 ppm
Jun 19 02:38:33 dist-test-slave-9tkz krb5kdc[26065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836713, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:33.277370 26078 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:38:33.278335 26078 webserver.cc:460] Webserver started at http://127.7.194.254:45251/ using document root <none> and password file <none>
I20260619 02:38:33.278904 26078 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:33.278999 26078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:33.279245 26078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:33.281358 26078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "38d8768830404bad8f3b2e64898d970d"
format_stamp: "Formatted at 2026-06-19 02:38:33 on dist-test-slave-9tkz"
server_key: "d7aa1b9bf1d8387221fe0d2ebc491bee"
server_key_iv: "f54979dd2f84bbfa17fe22c679b34411"
server_key_version: "encryptionkey@0"
I20260619 02:38:33.281981 26078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "38d8768830404bad8f3b2e64898d970d"
format_stamp: "Formatted at 2026-06-19 02:38:33 on dist-test-slave-9tkz"
server_key: "d7aa1b9bf1d8387221fe0d2ebc491bee"
server_key_iv: "f54979dd2f84bbfa17fe22c679b34411"
server_key_version: "encryptionkey@0"
I20260619 02:38:33.285753 26078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:38:33.288278 26093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:33.289769 26078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:38:33.289911 26078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "38d8768830404bad8f3b2e64898d970d"
format_stamp: "Formatted at 2026-06-19 02:38:33 on dist-test-slave-9tkz"
server_key: "d7aa1b9bf1d8387221fe0d2ebc491bee"
server_key_iv: "f54979dd2f84bbfa17fe22c679b34411"
server_key_version: "encryptionkey@0"
I20260619 02:38:33.290030 26078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:33.304641 26078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:33.307837 26078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:33.308053 26078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:33.315634 26078 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:38915
I20260619 02:38:33.315657 26145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:38915 every 8 connection(s)
I20260619 02:38:33.316644 26078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:33.319557 26146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:33.322715 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 26078
I20260619 02:38:33.322883 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:33.323176 7947 external_mini_cluster.cc:1428] Setting key fd8031b1dbf212580bd42704966331c4
I20260619 02:38:33.325654 26146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d: Bootstrap starting.
I20260619 02:38:33.327884 26146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:33.328625 26146 log.cc:824] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:33.330587 26146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d: No bootstrap required, opened a new log
Jun 19 02:38:33 dist-test-slave-9tkz krb5kdc[26065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:33.334461 26146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d8768830404bad8f3b2e64898d970d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38915 } }
I20260619 02:38:33.334710 26146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:33.334794 26146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d8768830404bad8f3b2e64898d970d, State: Initialized, Role: FOLLOWER
I20260619 02:38:33.335251 26146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d8768830404bad8f3b2e64898d970d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38915 } }
I20260619 02:38:33.335403 26146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:33.335486 26146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:33.335593 26146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:33.336818 26146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d8768830404bad8f3b2e64898d970d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38915 } }
I20260619 02:38:33.337183 26146 leader_election.cc:302] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38d8768830404bad8f3b2e64898d970d; no voters:
I20260619 02:38:33.337512 26146 leader_election.cc:288] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:33.337674 26151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:33.337910 26151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [term 1 LEADER]: Becoming Leader. State: Replica: 38d8768830404bad8f3b2e64898d970d, State: Running, Role: LEADER
I20260619 02:38:33.337980 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:33.324650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.7.194.254:38915)
0619 02:38:33.325110 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:33.325126 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:33.325164 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:38:33.325924 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:33.325944 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:33.326289 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:33.326630 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:33.327471 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.328547 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.329487 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.330032 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.330209 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:33.332718 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:33.332723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:33.332737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:33.332764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:33.334514 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:33.335011 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:33.335017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:33.335042 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:33.335186 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:33.335532 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:33.335536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:33.335538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:33.336077 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:33.336290 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:33.336594 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 02:38:33.336927 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":143,"threads_started":1}
I20260619 02:38:33.338307 26151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d8768830404bad8f3b2e64898d970d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38915 } }
I20260619 02:38:33.338824 26146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:33.339973 26153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38d8768830404bad8f3b2e64898d970d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d8768830404bad8f3b2e64898d970d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38915 } } }
I20260619 02:38:33.340099 26153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:33.340332 26153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38d8768830404bad8f3b2e64898d970d. Latest consensus state: current_term: 1 leader_uuid: "38d8768830404bad8f3b2e64898d970d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d8768830404bad8f3b2e64898d970d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38915 } } }
I20260619 02:38:33.340404 26153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:33.340801 26160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:33.343429 26160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:33.348798 26160 catalog_manager.cc:1347] Generated new cluster ID: 1c449e577be74caca1141c490a8ee913
I20260619 02:38:33.348871 26160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:33.365914 26160 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:33.367233 26160 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:33.377497 26160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d: Generated new TSK 0
I20260619 02:38:33.378290 26160 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:38:33.410744 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38915
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:33.523286 26172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:33.523562 26172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:33.523604 26172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:33.527657 26172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:33.527716 26172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:33.527776 26172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:38:33.533057 26172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:33.534144 26172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:33.535178 26172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:33.545179 26180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:33.545202 26178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:33.545202 26177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:33.545250 26172 server_base.cc:1034] running on GCE node
I20260619 02:38:33.545867 26172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:33.546645 26172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:33.547832 26172 hybrid_clock.cc:648] HybridClock initialized: now 1781836713547816 us; error 32 us; skew 500 ppm
Jun 19 02:38:33 dist-test-slave-9tkz krb5kdc[26065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836713, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:33.556150 26172 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:38:33.557161 26172 webserver.cc:460] Webserver started at http://127.7.194.193:45837/ using document root <none> and password file <none>
I20260619 02:38:33.557821 26172 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:33.557919 26172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:33.558158 26172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:33.560285 26172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "028368ff94194e2bbc98ee2bf9663a11"
format_stamp: "Formatted at 2026-06-19 02:38:33 on dist-test-slave-9tkz"
server_key: "05a762e990bcf6af58ba1a86265b55a9"
server_key_iv: "d3a68a7bd2397bebe3b116b7079149f3"
server_key_version: "encryptionkey@0"
I20260619 02:38:33.560880 26172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "028368ff94194e2bbc98ee2bf9663a11"
format_stamp: "Formatted at 2026-06-19 02:38:33 on dist-test-slave-9tkz"
server_key: "05a762e990bcf6af58ba1a86265b55a9"
server_key_iv: "d3a68a7bd2397bebe3b116b7079149f3"
server_key_version: "encryptionkey@0"
I20260619 02:38:33.564638 26172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:38:33.567322 26187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:33.568598 26172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:38:33.568768 26172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "028368ff94194e2bbc98ee2bf9663a11"
format_stamp: "Formatted at 2026-06-19 02:38:33 on dist-test-slave-9tkz"
server_key: "05a762e990bcf6af58ba1a86265b55a9"
server_key_iv: "d3a68a7bd2397bebe3b116b7079149f3"
server_key_version: "encryptionkey@0"
I20260619 02:38:33.568903 26172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:33.582707 26172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:33.586277 26172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:33.586505 26172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:33.587203 26172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:33.588068 26172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:33.588186 26172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:33.588285 26172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:33.588337 26172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:33.598567 26172 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:44033
I20260619 02:38:33.598594 26300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:44033 every 8 connection(s)
I20260619 02:38:33.599534 26172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:38:33.607507 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 26172
I20260619 02:38:33.607681 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:38:33.608064 7947 external_mini_cluster.cc:1428] Setting key 2f8d48c3ba96dc85729030ac0c717f83
Jun 19 02:38:33 dist-test-slave-9tkz krb5kdc[26065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836713, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:33.613598 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:33.601379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:42777 (local address 127.7.194.254:38915)
0619 02:38:33.601514 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:33.601517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:33.602431 (+ 914us) server_negotiation.cc:407] Connection header received
0619 02:38:33.603365 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:33.603369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:33.603430 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:33.603532 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:33.604762 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.605315 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.606563 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.606808 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.606925 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:33.609898 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:33.609902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:33.609904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:33.609928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:33.611425 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:33.611983 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:33.611986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:33.611987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:33.612063 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:33.612423 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:33.612426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:33.612428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:33.612615 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:33.612724 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:33.613258 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 02:38:33.613391 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:38:33.614454 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:33.601790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38915 (local address 127.7.194.193:42777)
0619 02:38:33.602267 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:33.602313 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:33.603132 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:33.603717 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:33.603724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:33.604101 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:33.604599 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:33.604613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.605444 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.605448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:33.606427 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:33.606437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.606965 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.606969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:33.607148 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:33.607810 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:33.607833 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:33.609692 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:33.611553 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:33.611559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:33.611577 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:33.611863 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:33.612156 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:33.612159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:33.612162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:33.612319 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:33.612734 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:33.612743 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:33.613006 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 02:38:33.613196 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 02:38:33.613574 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
I20260619 02:38:33.615897 26301 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38915
I20260619 02:38:33.616153 26301 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:33.616714 26301 heartbeater.cc:511] Master 127.7.194.254:38915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:38:33.618355 26110 ts_manager.cc:194] Registered new tserver with Master: 028368ff94194e2bbc98ee2bf9663a11 (127.7.194.193:44033)
I20260619 02:38:33.619445 26110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:42777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:38:33.636534 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38915
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:33.747740 26306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:33.748034 26306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:33.748080 26306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:33.752103 26306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:33.752169 26306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:33.752230 26306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:38:33.757507 26306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:33.758562 26306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:33.759608 26306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:33.769179 26311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:33.769325 26306 server_base.cc:1034] running on GCE node
W20260619 02:38:33.769143 26312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:33.769152 26314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:33.769848 26306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:33.770507 26306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:33.771713 26306 hybrid_clock.cc:648] HybridClock initialized: now 1781836713771677 us; error 50 us; skew 500 ppm
Jun 19 02:38:33 dist-test-slave-9tkz krb5kdc[26065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836713, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:33.774653 26306 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:38:33.775528 26306 webserver.cc:460] Webserver started at http://127.7.194.194:35821/ using document root <none> and password file <none>
I20260619 02:38:33.776111 26306 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:33.776172 26306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:33.776389 26306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:33.778693 26306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "a552a4b798704119a11094235a6e6a01"
format_stamp: "Formatted at 2026-06-19 02:38:33 on dist-test-slave-9tkz"
server_key: "9318851ea539a83cc068018873423304"
server_key_iv: "3c160b1f6b26cc0d4cc490c467011141"
server_key_version: "encryptionkey@0"
I20260619 02:38:33.779268 26306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "a552a4b798704119a11094235a6e6a01"
format_stamp: "Formatted at 2026-06-19 02:38:33 on dist-test-slave-9tkz"
server_key: "9318851ea539a83cc068018873423304"
server_key_iv: "3c160b1f6b26cc0d4cc490c467011141"
server_key_version: "encryptionkey@0"
I20260619 02:38:33.783121 26306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:38:33.785591 26321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:33.786935 26306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:38:33.787094 26306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "a552a4b798704119a11094235a6e6a01"
format_stamp: "Formatted at 2026-06-19 02:38:33 on dist-test-slave-9tkz"
server_key: "9318851ea539a83cc068018873423304"
server_key_iv: "3c160b1f6b26cc0d4cc490c467011141"
server_key_version: "encryptionkey@0"
I20260619 02:38:33.787217 26306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:33.812319 26306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:33.815726 26306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:33.815954 26306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:33.816615 26306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:33.817490 26306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:33.817586 26306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:33.817662 26306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:33.817698 26306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:33.826296 26306 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:44459
I20260619 02:38:33.826313 26434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:44459 every 8 connection(s)
I20260619 02:38:33.827198 26306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:38:33.833540 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 26306
I20260619 02:38:33.833719 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:38:33.834056 7947 external_mini_cluster.cc:1428] Setting key b932af348f138216ea422ba25968192e
Jun 19 02:38:33 dist-test-slave-9tkz krb5kdc[26065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836713, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:33.842892 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:33.829003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:40833 (local address 127.7.194.254:38915)
0619 02:38:33.829154 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:33.829159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:33.830172 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 02:38:33.831068 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:33.831071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:33.831117 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:33.831198 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:33.832415 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.832929 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.834709 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.834975 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.835078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:33.838349 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:33.838355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:33.838359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:33.838397 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:33.840305 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:33.840957 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:33.840963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:33.840968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:33.841100 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:33.841468 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:33.841475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:33.841481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:33.841814 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:33.841945 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:33.842462 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 02:38:33.842628 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:38:33.843502 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:33.829450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38915 (local address 127.7.194.194:40833)
0619 02:38:33.829994 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:33.830041 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:33.830871 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:33.831372 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:33.831379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:33.831774 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:33.832288 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:33.832303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.833036 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.833039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:33.834550 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:33.834560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:33.835130 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:33.835133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:33.835292 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:33.835975 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:33.836000 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:33.838135 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:33.840421 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:33.840426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:33.840439 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:33.840780 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:33.841210 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:33.841213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:33.841214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:33.841352 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:33.841933 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:33.841938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:33.842182 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 02:38:33.842340 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 02:38:33.842610 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":123,"threads_started":1}
I20260619 02:38:33.844738 26435 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38915
I20260619 02:38:33.844975 26435 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:33.845477 26435 heartbeater.cc:511] Master 127.7.194.254:38915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:38:33.846632 26110 ts_manager.cc:194] Registered new tserver with Master: a552a4b798704119a11094235a6e6a01 (127.7.194.194:44459)
I20260619 02:38:33.847498 26110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:40833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:38:33.863989 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38915
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:33.976925 26440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:33.977200 26440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:33.977299 26440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:33.981415 26440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:33.981480 26440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:33.981592 26440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:38:33.986963 26440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:38915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:33.988040 26440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:33.989068 26440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:38:33.999081 26440 server_base.cc:1034] running on GCE node
W20260619 02:38:33.999028 26445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:33.999104 26446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:33.999028 26448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:33.999779 26440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:34.000474 26440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:34.001626 26440 hybrid_clock.cc:648] HybridClock initialized: now 1781836714001605 us; error 29 us; skew 500 ppm
Jun 19 02:38:33 dist-test-slave-9tkz krb5kdc[26065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836713, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:34.004405 26440 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:38:34.005280 26440 webserver.cc:460] Webserver started at http://127.7.194.195:33469/ using document root <none> and password file <none>
I20260619 02:38:34.005909 26440 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:34.005970 26440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:34.006233 26440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:34.008263 26440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "bc9f387691da42cca23b92a365efdf7b"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "c3cfaa99f49cd1d81ebc5c620698f470"
server_key_iv: "e9b1c42c20927825e54f3cfdabb5f9ab"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.008807 26440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "bc9f387691da42cca23b92a365efdf7b"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "c3cfaa99f49cd1d81ebc5c620698f470"
server_key_iv: "e9b1c42c20927825e54f3cfdabb5f9ab"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.012630 26440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:38:34.015064 26455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.016247 26440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:38:34.016379 26440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "bc9f387691da42cca23b92a365efdf7b"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "c3cfaa99f49cd1d81ebc5c620698f470"
server_key_iv: "e9b1c42c20927825e54f3cfdabb5f9ab"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.016487 26440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:34.044847 26440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:34.047875 26440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:34.048096 26440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:34.048705 26440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:34.049574 26440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:34.049649 26440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.049722 26440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:34.049774 26440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.058871 26440 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:42823
I20260619 02:38:34.058890 26568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:42823 every 8 connection(s)
I20260619 02:38:34.059794 26440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:38:34.060601 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 26440
I20260619 02:38:34.060734 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:38:34.061085 7947 external_mini_cluster.cc:1428] Setting key e9e580b3deb6fbf2349676482cb2de5a
Jun 19 02:38:34 dist-test-slave-9tkz krb5kdc[26065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836713, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:34.073634 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.061975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:52873 (local address 127.7.194.254:38915)
0619 02:38:34.062102 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.062107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.063099 (+ 992us) server_negotiation.cc:407] Connection header received
0619 02:38:34.064213 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.064216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.064270 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.064347 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:34.065507 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.066086 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.067230 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.067470 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.067582 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.070075 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:34.070078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:34.070080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:34.070107 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:34.071553 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:34.072039 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:34.072043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:34.072044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:34.072163 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:34.072493 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:34.072496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:34.072497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:34.072748 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:34.072833 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.073340 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.073449 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:38:34.074844 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.062355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38915 (local address 127.7.194.195:52873)
0619 02:38:34.062931 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:34.063001 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:34.064015 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:34.064522 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:34.064529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:34.064893 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:34.065367 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.065381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.066210 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.066213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.067105 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.067113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.067623 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.067627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.067796 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.068428 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:34.068444 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:34.069904 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:34.071663 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:34.071667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:34.071679 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:34.071927 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:34.072251 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:34.072253 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:34.072254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:34.072382 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:34.072847 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:34.072853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:34.073089 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 02:38:34.073274 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 02:38:34.073889 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":166,"threads_started":1}
I20260619 02:38:34.076025 26569 heartbeater.cc:347] Connected to a master server at 127.7.194.254:38915
I20260619 02:38:34.076277 26569 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:34.076799 26569 heartbeater.cc:511] Master 127.7.194.254:38915 requested a full tablet report, sending...
I20260619 02:38:34.077901 26110 ts_manager.cc:194] Registered new tserver with Master: bc9f387691da42cca23b92a365efdf7b (127.7.194.195:42823)
I20260619 02:38:34.078507 26110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:52873
I20260619 02:38:34.086975 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:38:34.096920 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.088887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.7.194.254:38915)
0619 02:38:34.089035 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.089039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.089221 (+ 182us) server_negotiation.cc:407] Connection header received
0619 02:38:34.089370 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.089373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.089422 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.089523 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:34.090253 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.090834 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.091680 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.091968 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.092074 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.093651 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:34.093655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:34.093657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:34.093685 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:34.094966 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:34.095559 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:34.095562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:34.095564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:34.095662 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:34.095982 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:34.095986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:34.095987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:34.096213 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:34.096373 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.096641 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.096741 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:38:34.100034 26110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:38:34.103267 26110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:38:34.117902 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.112719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:42823 (local address 127.0.0.1:43456)
0619 02:38:34.112989 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:34.113025 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:34.113165 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:34.113989 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:34.113991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:34.114004 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:34.114144 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.114160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.115402 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.115408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.116525 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.116533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.117355 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.117358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.117461 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.117496 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:38:34.117633 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 02:38:34.117754 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":39,"thread_start_us":101,"threads_started":1}
I20260619 02:38:34.117913 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.112863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.7.194.195:42823)
0619 02:38:34.113494 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.113507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.113528 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:38:34.113630 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.113635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.113824 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.114299 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:34.114429 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.115275 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.116648 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.117231 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.117355 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.117504 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.117659 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.117754 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":131,"threads_started":1}
I20260619 02:38:34.118563 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.111916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:44033 (local address 127.0.0.1:39926)
0619 02:38:34.112967 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:34.113025 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:34.113171 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:34.113935 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:34.113939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:34.113969 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:34.114161 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.114167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.115150 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.115156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.116545 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.116553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.118118 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.118124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.118233 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.118256 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:38:34.118368 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:38:34.118436 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":60,"spinlock_wait_cycles":768,"thread_start_us":105,"threads_started":1}
I20260619 02:38:34.118682 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.112117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.7.194.193:44033)
0619 02:38:34.112456 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.112473 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.113115 (+ 642us) server_negotiation.cc:407] Connection header received
0619 02:38:34.113340 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.113346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.113597 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.113724 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:34.114279 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.115038 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.116722 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.117528 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.117670 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.117829 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.118410 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.118522 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":122,"threads_started":1}
I20260619 02:38:34.119486 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.111816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:44459 (local address 127.0.0.1:40026)
0619 02:38:34.113138 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:34.113166 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:34.113253 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:34.113919 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:34.113928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:34.113968 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:34.114154 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.114160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.116521 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.116524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.117685 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.117696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.118824 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.118828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.119215 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.119247 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 02:38:34.119298 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:38:34.119361 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":48,"spinlock_wait_cycles":1280,"thread_start_us":126,"threads_started":1}
I20260619 02:38:34.120231 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.112641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.7.194.194:44459)
0619 02:38:34.113452 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.113463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.113483 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:38:34.113575 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.113580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.113762 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.113901 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:34.115203 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.116338 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.117816 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.118671 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.119759 (+ 1088us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.119896 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.119944 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.120085 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":128,"threads_started":1}
I20260619 02:38:34.121244 26503 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb2e01e45704ebc828ed19a1b8c7df4 (DEFAULT_TABLE table=test-table [id=1722805eec0044419c42d17697cc5a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:34.121659 26235 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb2e01e45704ebc828ed19a1b8c7df4 (DEFAULT_TABLE table=test-table [id=1722805eec0044419c42d17697cc5a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:34.122323 26503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb2e01e45704ebc828ed19a1b8c7df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:34.122685 26235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb2e01e45704ebc828ed19a1b8c7df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:34.122805 26369 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb2e01e45704ebc828ed19a1b8c7df4 (DEFAULT_TABLE table=test-table [id=1722805eec0044419c42d17697cc5a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:34.123730 26369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb2e01e45704ebc828ed19a1b8c7df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:34.129060 26586 tablet_bootstrap.cc:492] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: Bootstrap starting.
I20260619 02:38:34.130415 26588 tablet_bootstrap.cc:492] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: Bootstrap starting.
I20260619 02:38:34.130853 26587 tablet_bootstrap.cc:492] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11: Bootstrap starting.
I20260619 02:38:34.131304 26586 tablet_bootstrap.cc:654] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:34.132315 26586 log.cc:824] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:34.132490 26588 tablet_bootstrap.cc:654] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:34.132887 26587 tablet_bootstrap.cc:654] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:34.133765 26588 log.cc:824] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:34.133765 26587 log.cc:824] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:34.134508 26586 tablet_bootstrap.cc:492] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: No bootstrap required, opened a new log
I20260619 02:38:34.134708 26586 ts_tablet_manager.cc:1397] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:38:34.135490 26588 tablet_bootstrap.cc:492] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: No bootstrap required, opened a new log
I20260619 02:38:34.135658 26587 tablet_bootstrap.cc:492] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11: No bootstrap required, opened a new log
I20260619 02:38:34.135700 26588 ts_tablet_manager.cc:1397] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:38:34.135833 26587 ts_tablet_manager.cc:1397] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 02:38:34.138549 26588 raft_consensus.cc:348] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } }
I20260619 02:38:34.138837 26588 raft_consensus.cc:374] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:34.138911 26588 raft_consensus.cc:729] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552a4b798704119a11094235a6e6a01, State: Initialized, Role: FOLLOWER
I20260619 02:38:34.138724 26586 raft_consensus.cc:348] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } }
I20260619 02:38:34.138980 26586 raft_consensus.cc:374] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:34.139042 26586 raft_consensus.cc:729] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9f387691da42cca23b92a365efdf7b, State: Initialized, Role: FOLLOWER
I20260619 02:38:34.139446 26588 consensus_queue.cc:260] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } }
I20260619 02:38:34.139523 26586 consensus_queue.cc:260] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } }
I20260619 02:38:34.140162 26435 heartbeater.cc:503] Master 127.7.194.254:38915 was elected leader, sending a full tablet report...
I20260619 02:38:34.140487 26588 ts_tablet_manager.cc:1428] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:38:34.140496 26586 ts_tablet_manager.cc:1428] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:38:34.140944 26569 heartbeater.cc:503] Master 127.7.194.254:38915 was elected leader, sending a full tablet report...
I20260619 02:38:34.141341 26587 raft_consensus.cc:348] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } }
I20260619 02:38:34.141517 26587 raft_consensus.cc:374] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:34.141600 26587 raft_consensus.cc:729] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028368ff94194e2bbc98ee2bf9663a11, State: Initialized, Role: FOLLOWER
I20260619 02:38:34.142030 26587 consensus_queue.cc:260] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } }
I20260619 02:38:34.142693 26301 heartbeater.cc:503] Master 127.7.194.254:38915 was elected leader, sending a full tablet report...
I20260619 02:38:34.142967 26587 ts_tablet_manager.cc:1428] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260619 02:38:34.219691 26594 raft_consensus.cc:482] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:34.219980 26594 raft_consensus.cc:504] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } }
I20260619 02:38:34.221156 26594 leader_election.cc:288] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a552a4b798704119a11094235a6e6a01 (127.7.194.194:44459), bc9f387691da42cca23b92a365efdf7b (127.7.194.195:42823)
I20260619 02:38:34.226671 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.221603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:51651 (local address 127.7.194.194:44459)
0619 02:38:34.221758 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.221763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.222100 (+ 337us) server_negotiation.cc:407] Connection header received
0619 02:38:34.222224 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.222228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.222286 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.222376 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:34.222787 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.223590 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.225355 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.226004 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.226132 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.226187 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.226462 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.226534 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:38:34.226667 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.221680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:52995 (local address 127.7.194.195:42823)
0619 02:38:34.221795 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.221799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.222049 (+ 250us) server_negotiation.cc:407] Connection header received
0619 02:38:34.222234 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.222237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.222281 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.222375 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:34.222787 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.223601 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.225355 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.226004 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.226132 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.226213 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.226462 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.226534 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:38:34.226732 26596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.221456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:44459 (local address 127.7.194.193:51651)
0619 02:38:34.222003 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:34.222018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:34.222106 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:34.222405 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:34.222408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:34.222447 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:34.222666 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.222671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.223729 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.223733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.225215 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.225227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.226138 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.226142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.226309 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.226352 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 02:38:34.226430 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:38:34.226536 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":278,"spinlock_wait_cycles":28160,"thread_start_us":155,"threads_started":1}
I20260619 02:38:34.226735 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.221456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:42823 (local address 127.7.194.193:52995)
0619 02:38:34.221937 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:34.221954 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:34.222098 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:34.222405 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:34.222408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:34.222448 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:34.222661 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.222668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.223729 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.223733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.225215 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.225227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.226138 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.226142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.226305 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.226349 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:38:34.226430 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:38:34.226536 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":135,"spinlock_wait_cycles":5504,"thread_start_us":176,"threads_started":1}
I20260619 02:38:34.227502 26389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb2e01e45704ebc828ed19a1b8c7df4" candidate_uuid: "028368ff94194e2bbc98ee2bf9663a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552a4b798704119a11094235a6e6a01" is_pre_election: true
I20260619 02:38:34.227618 26523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb2e01e45704ebc828ed19a1b8c7df4" candidate_uuid: "028368ff94194e2bbc98ee2bf9663a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9f387691da42cca23b92a365efdf7b" is_pre_election: true
I20260619 02:38:34.227867 26389 raft_consensus.cc:2454] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 028368ff94194e2bbc98ee2bf9663a11 in term 0.
I20260619 02:38:34.227963 26523 raft_consensus.cc:2454] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 028368ff94194e2bbc98ee2bf9663a11 in term 0.
I20260619 02:38:34.228469 26191 leader_election.cc:302] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 028368ff94194e2bbc98ee2bf9663a11, a552a4b798704119a11094235a6e6a01, bc9f387691da42cca23b92a365efdf7b; no voters:
I20260619 02:38:34.228758 26594 raft_consensus.cc:2781] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:38:34.228844 26594 raft_consensus.cc:482] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:38:34.228904 26594 raft_consensus.cc:3037] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:34.229957 26594 raft_consensus.cc:504] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } }
I20260619 02:38:34.230304 26594 leader_election.cc:288] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [CANDIDATE]: Term 1 election: Requested vote from peers a552a4b798704119a11094235a6e6a01 (127.7.194.194:44459), bc9f387691da42cca23b92a365efdf7b (127.7.194.195:42823)
I20260619 02:38:34.230645 26389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb2e01e45704ebc828ed19a1b8c7df4" candidate_uuid: "028368ff94194e2bbc98ee2bf9663a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552a4b798704119a11094235a6e6a01"
I20260619 02:38:34.230702 26523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb2e01e45704ebc828ed19a1b8c7df4" candidate_uuid: "028368ff94194e2bbc98ee2bf9663a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9f387691da42cca23b92a365efdf7b"
I20260619 02:38:34.230788 26389 raft_consensus.cc:3037] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:34.230820 26523 raft_consensus.cc:3037] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:34.232013 26523 raft_consensus.cc:2454] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 028368ff94194e2bbc98ee2bf9663a11 in term 1.
I20260619 02:38:34.231992 26389 raft_consensus.cc:2454] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 028368ff94194e2bbc98ee2bf9663a11 in term 1.
I20260619 02:38:34.232309 26191 leader_election.cc:302] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 028368ff94194e2bbc98ee2bf9663a11, a552a4b798704119a11094235a6e6a01; no voters:
I20260619 02:38:34.232486 26594 raft_consensus.cc:2781] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:34.232722 26594 raft_consensus.cc:686] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 1 LEADER]: Becoming Leader. State: Replica: 028368ff94194e2bbc98ee2bf9663a11, State: Running, Role: LEADER
I20260619 02:38:34.233099 26594 consensus_queue.cc:237] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } }
I20260619 02:38:34.236281 26109 catalog_manager.cc:5495] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 028368ff94194e2bbc98ee2bf9663a11 (127.7.194.193). New cstate: current_term: 1 leader_uuid: "028368ff94194e2bbc98ee2bf9663a11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "028368ff94194e2bbc98ee2bf9663a11" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 } health_report { overall_health: UNKNOWN } } }
I20260619 02:38:34.271029 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.267404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.7.194.193:44033)
0619 02:38:34.267565 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.267570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.267585 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:38:34.267701 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.267703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.267757 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.267869 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:38:34.268119 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.268619 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.269438 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.269718 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.269827 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.270178 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:38:34.270598 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:38:34.270763 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.270840 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.270879 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:38:34.277144 26389 raft_consensus.cc:1261] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [term 1 FOLLOWER]: Refusing update from remote peer 028368ff94194e2bbc98ee2bf9663a11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:34.277143 26523 raft_consensus.cc:1261] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [term 1 FOLLOWER]: Refusing update from remote peer 028368ff94194e2bbc98ee2bf9663a11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:34.277894 26594 consensus_queue.cc:1035] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc9f387691da42cca23b92a365efdf7b" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 42823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:34.278122 26597 consensus_queue.cc:1035] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552a4b798704119a11094235a6e6a01" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:34.289335 26602 mvcc.cc:205] Tried to move back new op lower bound from 7298403181672116224 to 7298403181501878272. Current Snapshot: MvccSnapshot[applied={T|T < 7298403181672116224 or (T in {7298403181672116224})}]
I20260619 02:38:34.295258 26110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:34.295459 26110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:34.298069 26110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38d8768830404bad8f3b2e64898d970d: Sending DeleteTablet for 3 replicas of tablet 8cb2e01e45704ebc828ed19a1b8c7df4
I20260619 02:38:34.298890 26369 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb2e01e45704ebc828ed19a1b8c7df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:34 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:40026
I20260619 02:38:34.299104 26235 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb2e01e45704ebc828ed19a1b8c7df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:34 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:39926
I20260619 02:38:34.299470 26610 tablet_replica.cc:330] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: stopping tablet replica
I20260619 02:38:34.299628 26611 tablet_replica.cc:330] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11: stopping tablet replica
I20260619 02:38:34.299641 26503 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb2e01e45704ebc828ed19a1b8c7df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:34 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:43456
I20260619 02:38:34.299860 26610 raft_consensus.cc:2229] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:34.299893 26611 raft_consensus.cc:2229] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:38:34.300165 26610 raft_consensus.cc:2258] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:34.300256 26611 raft_consensus.cc:2258] T 8cb2e01e45704ebc828ed19a1b8c7df4 P 028368ff94194e2bbc98ee2bf9663a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:34.300966 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 26172
I20260619 02:38:34.303196 26610 ts_tablet_manager.cc:1905] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:34.305346 26610 ts_tablet_manager.cc:1918] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:34.305456 26610 log.cc:1196] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/8cb2e01e45704ebc828ed19a1b8c7df4
I20260619 02:38:34.305789 26612 tablet_replica.cc:330] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: stopping tablet replica
I20260619 02:38:34.305845 26610 ts_tablet_manager.cc:1939] T 8cb2e01e45704ebc828ed19a1b8c7df4 P a552a4b798704119a11094235a6e6a01: Deleting consensus metadata
I20260619 02:38:34.306137 26612 raft_consensus.cc:2229] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:34.306453 26612 raft_consensus.cc:2258] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:34.307066 26097 catalog_manager.cc:4841] TS a552a4b798704119a11094235a6e6a01 (127.7.194.194:44459): tablet 8cb2e01e45704ebc828ed19a1b8c7df4 (table test-table [id=1722805eec0044419c42d17697cc5a70]) successfully deleted
W20260619 02:38:34.307463 26095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:44033 (error 108)
I20260619 02:38:34.307945 26612 ts_tablet_manager.cc:1905] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:34.308439 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.308138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:44033 (local address 127.0.0.1:39942)
0619 02:38:34.308267 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:34.308345 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:38:34.308523 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 26306
W20260619 02:38:34.309664 26095 catalog_manager.cc:4584] TS 028368ff94194e2bbc98ee2bf9663a11 (127.7.194.193:44033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cb2e01e45704ebc828ed19a1b8c7df4: Network error: Client connection negotiation failed: client connection to 127.7.194.193:44033: connect: Connection refused (error 111)
I20260619 02:38:34.311126 26612 ts_tablet_manager.cc:1918] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:34.311228 26612 log.cc:1196] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/8cb2e01e45704ebc828ed19a1b8c7df4
I20260619 02:38:34.311538 26612 ts_tablet_manager.cc:1939] T 8cb2e01e45704ebc828ed19a1b8c7df4 P bc9f387691da42cca23b92a365efdf7b: Deleting consensus metadata
I20260619 02:38:34.312817 26096 catalog_manager.cc:4841] TS bc9f387691da42cca23b92a365efdf7b (127.7.194.195:42823): tablet 8cb2e01e45704ebc828ed19a1b8c7df4 (table test-table [id=1722805eec0044419c42d17697cc5a70]) successfully deleted
I20260619 02:38:34.314898 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 26440
I20260619 02:38:34.322245 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 26078
2026-06-19T02:38:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2351 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T02:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:34Z Disabled control of system clock
I20260619 02:38:34.352836 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44721
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:44721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:38:34.465592 26617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:34.465844 26617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:34.465881 26617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:34.470077 26617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:34.470130 26617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:34.470146 26617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:34.470163 26617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:34.470218 26617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:34.474939 26617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:44721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:34.476027 26617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:34.477455 26617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:34.485944 26623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:34.486024 26622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:34.486056 26625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:34.486342 26617 server_base.cc:1034] running on GCE node
I20260619 02:38:34.486873 26617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:34.487931 26617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:34.489104 26617 hybrid_clock.cc:648] HybridClock initialized: now 1781836714489067 us; error 43 us; skew 500 ppm
I20260619 02:38:34.490995 26617 webserver.cc:460] Webserver started at http://127.7.194.254:32879/ using document root <none> and password file <none>
I20260619 02:38:34.491590 26617 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:34.491650 26617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:34.491860 26617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:34.493989 26617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "1863f6e63f15418c817db90342b07979"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "301fe33cd206302858ab997ea9e17518"
server_key_iv: "949401b15d6ac7613736612990b76572"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.494529 26617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "1863f6e63f15418c817db90342b07979"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "301fe33cd206302858ab997ea9e17518"
server_key_iv: "949401b15d6ac7613736612990b76572"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.498157 26617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:38:34.500571 26631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.501974 26617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:34.502120 26617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "1863f6e63f15418c817db90342b07979"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "301fe33cd206302858ab997ea9e17518"
server_key_iv: "949401b15d6ac7613736612990b76572"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.502246 26617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:34.526542 26617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:34.527346 26617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:34.527560 26617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:34.534880 26617 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:44721
I20260619 02:38:34.534886 26683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:44721 every 8 connection(s)
I20260619 02:38:34.535913 26617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:34.538960 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 26617
I20260619 02:38:34.539093 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:34.539433 7947 external_mini_cluster.cc:1428] Setting key 1a35c916f82c1a027281b35483cb5f32
I20260619 02:38:34.539472 26684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:34.546811 26684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979: Bootstrap starting.
I20260619 02:38:34.549377 26684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:34.549407 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.540856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.7.194.254:44721)
0619 02:38:34.541501 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.541517 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.541578 (+ 61us) server_negotiation.cc:407] Connection header received
0619 02:38:34.542242 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.542261 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.542676 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.542957 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:34.543332 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.544335 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.545670 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.546195 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.546330 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.547395 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:34.547400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:34.547413 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:34.547812 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:34.547865 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:34.547885 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:34.547962 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.548148 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.548451 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
I20260619 02:38:34.550330 26684 log.cc:824] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:34.552466 26684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979: No bootstrap required, opened a new log
I20260619 02:38:34.556114 26684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1863f6e63f15418c817db90342b07979" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44721 } }
I20260619 02:38:34.556351 26684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:34.556432 26684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1863f6e63f15418c817db90342b07979, State: Initialized, Role: FOLLOWER
I20260619 02:38:34.556840 26684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1863f6e63f15418c817db90342b07979" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44721 } }
I20260619 02:38:34.556993 26684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:34.557060 26684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:34.557168 26684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:34.558267 26684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1863f6e63f15418c817db90342b07979" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44721 } }
I20260619 02:38:34.558609 26684 leader_election.cc:302] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1863f6e63f15418c817db90342b07979; no voters:
I20260619 02:38:34.558925 26684 leader_election.cc:288] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:34.559069 26689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:34.559333 26689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [term 1 LEADER]: Becoming Leader. State: Replica: 1863f6e63f15418c817db90342b07979, State: Running, Role: LEADER
I20260619 02:38:34.559669 26689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1863f6e63f15418c817db90342b07979" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44721 } }
I20260619 02:38:34.560122 26684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:34.561348 26691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1863f6e63f15418c817db90342b07979. Latest consensus state: current_term: 1 leader_uuid: "1863f6e63f15418c817db90342b07979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1863f6e63f15418c817db90342b07979" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44721 } } }
I20260619 02:38:34.561475 26691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:34.562129 26698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:34.562095 26690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1863f6e63f15418c817db90342b07979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1863f6e63f15418c817db90342b07979" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44721 } } }
I20260619 02:38:34.562242 26690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:34.564967 26698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:34.570430 26698 catalog_manager.cc:1347] Generated new cluster ID: 0e6c7fcd7f834639af1cf0af6df9e816
I20260619 02:38:34.570508 26698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:34.582881 26698 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:34.584280 26698 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:34.604821 26698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979: Generated new TSK 0
I20260619 02:38:34.605803 26698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:34.613870 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44721
--builtin_ntp_servers=127.7.194.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:38:34.727399 26708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:34.727674 26708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:34.727710 26708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:34.731748 26708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:34.731801 26708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:34.731877 26708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:38:34.736629 26708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:34.737790 26708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:34.739207 26708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:34.749380 26714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:34.749337 26713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:34.749583 26708 server_base.cc:1034] running on GCE node
W20260619 02:38:34.749337 26716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:34.750224 26708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:34.750816 26708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:34.751987 26708 hybrid_clock.cc:648] HybridClock initialized: now 1781836714751953 us; error 43 us; skew 500 ppm
I20260619 02:38:34.753968 26708 webserver.cc:460] Webserver started at http://127.7.194.193:44403/ using document root <none> and password file <none>
I20260619 02:38:34.754515 26708 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:34.754575 26708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:34.754746 26708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:34.756728 26708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "83fd2a4c3c19f605bec56f42fd696474"
server_key_iv: "d8216fdf3d3f2451e6288be3dbccd430"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.757220 26708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "83fd2a4c3c19f605bec56f42fd696474"
server_key_iv: "d8216fdf3d3f2451e6288be3dbccd430"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.761071 26708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:38:34.763651 26722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.764884 26708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:34.764983 26708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "83fd2a4c3c19f605bec56f42fd696474"
server_key_iv: "d8216fdf3d3f2451e6288be3dbccd430"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.765079 26708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:34.782510 26708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:34.783143 26708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:34.783324 26708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:34.783918 26708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:34.784823 26708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:34.784871 26708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.784912 26708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:34.784931 26708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.793799 26708 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:41479
I20260619 02:38:34.793835 26835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:41479 every 8 connection(s)
I20260619 02:38:34.794693 26708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:38:34.800374 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 26708
I20260619 02:38:34.800516 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:38:34.800833 7947 external_mini_cluster.cc:1428] Setting key a9d700661633dc2f94ef4568d7434e5e
I20260619 02:38:34.803048 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44721
--builtin_ntp_servers=127.7.194.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:38:34.804641 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.796514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:42261 (local address 127.7.194.254:44721)
0619 02:38:34.796656 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.796660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.797770 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 02:38:34.799174 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.799178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.799234 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.799324 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:34.800284 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.800950 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.802463 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.802691 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.802772 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.804046 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:34.804049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:34.804052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:34.804063 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:34.804090 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:34.804098 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:34.804166 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.804364 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.804453 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:38:34.805528 26838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.796919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44721 (local address 127.7.194.193:42261)
0619 02:38:34.797617 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:34.797663 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:34.798955 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:34.799507 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:34.799515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:34.799596 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:34.800102 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.800121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.801095 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.801098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.802324 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.802333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.802851 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.802859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.803078 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.803743 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:38:34.803747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:34.803803 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:38:34.803889 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:34.804191 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:34.804202 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:34.804211 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:38:34.804337 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 02:38:34.804651 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260619 02:38:34.806910 26836 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44721
I20260619 02:38:34.807138 26836 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:34.807682 26836 heartbeater.cc:511] Master 127.7.194.254:44721 requested a full tablet report, sending...
I20260619 02:38:34.809391 26647 ts_manager.cc:194] Registered new tserver with Master: ff25f75bac6f46a8a89bd4e7f1d9de51 (127.7.194.193:41479)
I20260619 02:38:34.811182 26647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.193:42261
W20260619 02:38:34.916438 26839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:34.916702 26839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:34.916741 26839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:34.920710 26839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:34.920758 26839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:34.920876 26839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:38:34.925503 26839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:34.926654 26839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:34.928148 26839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:34.937975 26844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:34.938030 26847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:34.937983 26845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:34.938292 26839 server_base.cc:1034] running on GCE node
I20260619 02:38:34.938702 26839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:34.939352 26839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:34.940541 26839 hybrid_clock.cc:648] HybridClock initialized: now 1781836714940529 us; error 30 us; skew 500 ppm
I20260619 02:38:34.942492 26839 webserver.cc:460] Webserver started at http://127.7.194.194:41435/ using document root <none> and password file <none>
I20260619 02:38:34.943092 26839 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:34.943192 26839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:34.943420 26839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:34.945607 26839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "1c6adb26eef14696becee1ae568558fc"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "beac5c3b7f0c3f6810b4f448343c1cd8"
server_key_iv: "c15f4a111541cd92942de380f6677bf0"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.946197 26839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "1c6adb26eef14696becee1ae568558fc"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "beac5c3b7f0c3f6810b4f448343c1cd8"
server_key_iv: "c15f4a111541cd92942de380f6677bf0"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.950150 26839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:38:34.952906 26853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.954442 26839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:38:34.954586 26839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "1c6adb26eef14696becee1ae568558fc"
format_stamp: "Formatted at 2026-06-19 02:38:34 on dist-test-slave-9tkz"
server_key: "beac5c3b7f0c3f6810b4f448343c1cd8"
server_key_iv: "c15f4a111541cd92942de380f6677bf0"
server_key_version: "encryptionkey@0"
I20260619 02:38:34.954720 26839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:34.970407 26839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:34.971153 26839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:34.971369 26839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:34.972011 26839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:34.973024 26839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:34.973101 26839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.973183 26839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:34.973230 26839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:34.982843 26839 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:45679
I20260619 02:38:34.982865 26966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:45679 every 8 connection(s)
I20260619 02:38:34.983888 26839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:38:34.990505 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 26839
I20260619 02:38:34.990614 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:38:34.990917 7947 external_mini_cluster.cc:1428] Setting key 94867611552615423a9ede621e1636f2
I20260619 02:38:34.993178 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44721
--builtin_ntp_servers=127.7.194.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:38:34.994294 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.985880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:45335 (local address 127.7.194.254:44721)
0619 02:38:34.986063 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:34.986069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:34.987214 (+ 1145us) server_negotiation.cc:407] Connection header received
0619 02:38:34.988635 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:34.988650 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:34.988698 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:34.988779 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:34.989779 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.990364 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.991942 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.992251 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.992367 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.993680 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:34.993688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:34.993695 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:34.993713 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:34.993742 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:34.993757 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:34.993845 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:34.993982 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 02:38:34.994057 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":84}
I20260619 02:38:34.995062 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:34.986377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44721 (local address 127.7.194.194:45335)
0619 02:38:34.987043 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:34.987083 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:34.988401 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:34.988939 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:34.988946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:34.989028 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:34.989626 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.989651 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.990511 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.990515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.991812 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:34.991821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:34.992429 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:34.992433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:34.992636 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:34.993329 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:38:34.993331 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:34.993389 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:38:34.993481 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:34.993845 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:34.993852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:34.993856 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:38:34.993953 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 02:38:34.994217 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260619 02:38:34.996248 26967 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44721
I20260619 02:38:34.996496 26967 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:34.997006 26967 heartbeater.cc:511] Master 127.7.194.254:44721 requested a full tablet report, sending...
I20260619 02:38:34.998145 26647 ts_manager.cc:194] Registered new tserver with Master: 1c6adb26eef14696becee1ae568558fc (127.7.194.194:45679)
I20260619 02:38:34.999099 26647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.194:45335
W20260619 02:38:35.109836 26970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:35.110100 26970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:35.110158 26970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:35.114113 26970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:35.114167 26970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:35.114223 26970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:38:35.118862 26970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:44721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:35.119982 26970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:35.121366 26970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:35.131035 26978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:35.131057 26970 server_base.cc:1034] running on GCE node
W20260619 02:38:35.131075 26976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:35.131057 26975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:35.131649 26970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:35.132310 26970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:35.133514 26970 hybrid_clock.cc:648] HybridClock initialized: now 1781836715133471 us; error 55 us; skew 500 ppm
I20260619 02:38:35.135517 26970 webserver.cc:460] Webserver started at http://127.7.194.195:34507/ using document root <none> and password file <none>
I20260619 02:38:35.136092 26970 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:35.136209 26970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:35.136457 26970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:35.138523 26970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "04f38b859b9a4c2298cfd5cb801c1f5c"
format_stamp: "Formatted at 2026-06-19 02:38:35 on dist-test-slave-9tkz"
server_key: "1f146616af2fd2ab6222135722f12b5f"
server_key_iv: "b591e42395efa40a2add27886367a572"
server_key_version: "encryptionkey@0"
I20260619 02:38:35.139060 26970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "04f38b859b9a4c2298cfd5cb801c1f5c"
format_stamp: "Formatted at 2026-06-19 02:38:35 on dist-test-slave-9tkz"
server_key: "1f146616af2fd2ab6222135722f12b5f"
server_key_iv: "b591e42395efa40a2add27886367a572"
server_key_version: "encryptionkey@0"
I20260619 02:38:35.142769 26970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:38:35.145095 26984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:35.146451 26970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:35.146582 26970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "04f38b859b9a4c2298cfd5cb801c1f5c"
format_stamp: "Formatted at 2026-06-19 02:38:35 on dist-test-slave-9tkz"
server_key: "1f146616af2fd2ab6222135722f12b5f"
server_key_iv: "b591e42395efa40a2add27886367a572"
server_key_version: "encryptionkey@0"
I20260619 02:38:35.146692 26970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:35.169644 26970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:35.170389 26970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:35.170586 26970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:35.171188 26970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:35.172159 26970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:35.172231 26970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:35.172304 26970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:35.172349 26970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:35.181459 26970 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:44483
I20260619 02:38:35.181509 27097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:44483 every 8 connection(s)
I20260619 02:38:35.182475 26970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:38:35.190382 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 26970
I20260619 02:38:35.190488 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:38:35.190879 7947 external_mini_cluster.cc:1428] Setting key 353e4c3c8505f8814808397d08db0175
I20260619 02:38:35.192147 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.184450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:33755 (local address 127.7.194.254:44721)
0619 02:38:35.184583 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:35.184587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:35.185684 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 02:38:35.187107 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:35.187111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:35.187171 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:35.187253 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:35.188194 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.188735 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.189914 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.190140 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.190254 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.191563 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:35.191583 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:35.191586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:35.191602 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:35.191628 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:35.191642 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:35.191724 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:35.191922 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 02:38:35.191992 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:38:35.193099 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.184827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:44721 (local address 127.7.194.195:33755)
0619 02:38:35.185478 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:35.185519 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:35.186885 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:35.187418 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:35.187425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:35.187510 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:35.188014 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.188027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.188863 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.188867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.189781 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.189790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.190292 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.190296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.190497 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.191268 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:38:35.191270 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:35.191322 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:38:35.191406 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:35.191746 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:35.191754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:35.191761 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:38:35.191904 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 02:38:35.192174 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260619 02:38:35.194270 27098 heartbeater.cc:347] Connected to a master server at 127.7.194.254:44721
I20260619 02:38:35.194505 27098 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:35.195044 27098 heartbeater.cc:511] Master 127.7.194.254:44721 requested a full tablet report, sending...
I20260619 02:38:35.196121 26647 ts_manager.cc:194] Registered new tserver with Master: 04f38b859b9a4c2298cfd5cb801c1f5c (127.7.194.195:44483)
I20260619 02:38:35.196734 26647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.195:33755
I20260619 02:38:35.205320 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:38:35.211975 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.207600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.7.194.254:44721)
0619 02:38:35.207765 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:35.207771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:35.207930 (+ 159us) server_negotiation.cc:407] Connection header received
0619 02:38:35.208142 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:35.208144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:35.208188 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:35.208285 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:35.208580 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.209161 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.210167 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.210417 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.210547 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.211396 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:35.211400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:35.211405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:35.211422 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:35.211440 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:35.211447 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:35.211582 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:35.211751 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 02:38:35.211792 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:38:35.215031 26647 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:38:35.218312 26647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:38:35.232993 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.227017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.7.194.194:45679)
0619 02:38:35.227465 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:35.227478 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:35.227857 (+ 379us) server_negotiation.cc:407] Connection header received
0619 02:38:35.227940 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:35.227946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:35.228213 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:35.228373 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:35.229027 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.229993 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.231645 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.232318 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.232437 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.232632 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:35.232731 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 02:38:35.232826 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":121,"threads_started":1}
I20260619 02:38:35.232991 27109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.226901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:45679 (local address 127.0.0.1:39520)
0619 02:38:35.227499 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:35.227546 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:35.227698 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:35.228551 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:35.228557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:35.228597 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:35.228875 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.228883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.230100 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.230104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.231514 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.231526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.232439 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.232442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.232549 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.232587 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:38:35.232707 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 02:38:35.232832 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":92,"spinlock_wait_cycles":8192,"thread_start_us":76,"threads_started":1}
I20260619 02:38:35.233316 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.226975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:44483 (local address 127.0.0.1:44860)
0619 02:38:35.227753 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:35.227766 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:35.227856 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:35.228551 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:35.228557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:35.228608 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:35.228875 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.228883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.230350 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.230354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.231551 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.231561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.232706 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.232710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.232808 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.232837 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:38:35.232898 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 02:38:35.232966 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":148,"spinlock_wait_cycles":42240,"thread_start_us":98,"threads_started":1}
I20260619 02:38:35.233623 27114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.228021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60326 (local address 127.7.194.193:41479)
0619 02:38:35.228840 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:35.228851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:35.228868 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:38:35.228927 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:35.228930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:35.229144 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:35.229294 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:35.229511 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.230534 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.232024 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.232754 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.232849 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.233020 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:35.233320 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 02:38:35.233433 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":170,"threads_started":1}
I20260619 02:38:35.233675 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.227582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.7.194.195:44483)
0619 02:38:35.228004 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:35.228014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:35.228033 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:38:35.228095 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:35.228099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:35.228265 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:35.228389 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:35.229019 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.230234 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.231680 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.232371 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.232470 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.232643 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:35.233060 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 02:38:35.233458 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":97,"threads_started":1}
I20260619 02:38:35.233457 27113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.227615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41479 (local address 127.0.0.1:60326)
0619 02:38:35.228484 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:35.228497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:35.228603 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:35.229292 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:35.229295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:35.229314 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:35.229393 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.229399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.230663 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.230666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.231896 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.231906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.233077 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.233080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.233167 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.233191 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:38:35.233263 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:38:35.233327 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":80,"spinlock_wait_cycles":18688,"thread_start_us":79,"threads_started":1}
I20260619 02:38:35.236367 26901 tablet_service.cc:1467] Processing CreateTablet for tablet 2af0468f4203476e92bd50f5838267e3 (DEFAULT_TABLE table=test-table [id=e250e219c5384192a315a2ff573ee959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:35.236471 27032 tablet_service.cc:1467] Processing CreateTablet for tablet 2af0468f4203476e92bd50f5838267e3 (DEFAULT_TABLE table=test-table [id=e250e219c5384192a315a2ff573ee959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:35.236471 26770 tablet_service.cc:1467] Processing CreateTablet for tablet 2af0468f4203476e92bd50f5838267e3 (DEFAULT_TABLE table=test-table [id=e250e219c5384192a315a2ff573ee959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:35.237388 26901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af0468f4203476e92bd50f5838267e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:35.237468 26770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af0468f4203476e92bd50f5838267e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:35.237468 27032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af0468f4203476e92bd50f5838267e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:35.244098 27115 tablet_bootstrap.cc:492] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: Bootstrap starting.
I20260619 02:38:35.245316 27116 tablet_bootstrap.cc:492] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51: Bootstrap starting.
I20260619 02:38:35.245877 27117 tablet_bootstrap.cc:492] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: Bootstrap starting.
I20260619 02:38:35.246428 27115 tablet_bootstrap.cc:654] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:35.247164 27115 log.cc:824] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:35.247509 27116 tablet_bootstrap.cc:654] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:35.247876 27117 tablet_bootstrap.cc:654] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:35.248365 27116 log.cc:824] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:35.248683 27117 log.cc:824] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:35.249291 27115 tablet_bootstrap.cc:492] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: No bootstrap required, opened a new log
I20260619 02:38:35.249495 27115 ts_tablet_manager.cc:1397] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:38:35.250598 27116 tablet_bootstrap.cc:492] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51: No bootstrap required, opened a new log
I20260619 02:38:35.250794 27116 ts_tablet_manager.cc:1397] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:38:35.251376 27117 tablet_bootstrap.cc:492] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: No bootstrap required, opened a new log
I20260619 02:38:35.251578 27117 ts_tablet_manager.cc:1397] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 02:38:35.253628 27115 raft_consensus.cc:348] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } }
I20260619 02:38:35.253902 27115 raft_consensus.cc:374] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:35.253959 27115 raft_consensus.cc:729] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6adb26eef14696becee1ae568558fc, State: Initialized, Role: FOLLOWER
I20260619 02:38:35.254441 27115 consensus_queue.cc:260] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } }
I20260619 02:38:35.255292 26967 heartbeater.cc:503] Master 127.7.194.254:44721 was elected leader, sending a full tablet report...
I20260619 02:38:35.255637 27115 ts_tablet_manager.cc:1428] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:38:35.255685 27117 raft_consensus.cc:348] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } }
I20260619 02:38:35.255916 27117 raft_consensus.cc:374] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:35.255988 27117 raft_consensus.cc:729] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f38b859b9a4c2298cfd5cb801c1f5c, State: Initialized, Role: FOLLOWER
I20260619 02:38:35.256433 27117 consensus_queue.cc:260] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } }
I20260619 02:38:35.256600 27116 raft_consensus.cc:348] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } }
I20260619 02:38:35.256848 27116 raft_consensus.cc:374] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:35.256906 27116 raft_consensus.cc:729] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff25f75bac6f46a8a89bd4e7f1d9de51, State: Initialized, Role: FOLLOWER
I20260619 02:38:35.257283 27098 heartbeater.cc:503] Master 127.7.194.254:44721 was elected leader, sending a full tablet report...
I20260619 02:38:35.257351 27116 consensus_queue.cc:260] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } }
I20260619 02:38:35.258515 27117 ts_tablet_manager.cc:1428] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 02:38:35.258515 27116 ts_tablet_manager.cc:1428] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 02:38:35.258951 26836 heartbeater.cc:503] Master 127.7.194.254:44721 was elected leader, sending a full tablet report...
W20260619 02:38:35.295878 26837 tablet.cc:2369] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:38:35.433497 27099 tablet.cc:2369] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:38:35.485146 26968 tablet.cc:2369] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:38:35.619742 27123 raft_consensus.cc:482] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:35.619990 27123 raft_consensus.cc:504] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } }
I20260619 02:38:35.621254 27123 leader_election.cc:288] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c6adb26eef14696becee1ae568558fc (127.7.194.194:45679), 04f38b859b9a4c2298cfd5cb801c1f5c (127.7.194.195:44483)
I20260619 02:38:35.625339 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.621688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:43597 (local address 127.7.194.194:45679)
0619 02:38:35.621824 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:35.621828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:35.622012 (+ 184us) server_negotiation.cc:407] Connection header received
0619 02:38:35.622184 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:35.622190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:35.622240 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:35.622350 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:35.622685 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.623176 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.624330 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.624738 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.624868 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.624918 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:35.625111 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:38:35.625189 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:38:35.625356 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.621737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:56623 (local address 127.7.194.195:44483)
0619 02:38:35.621970 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:35.621975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:35.622107 (+ 132us) server_negotiation.cc:407] Connection header received
0619 02:38:35.622260 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:35.622265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:35.622309 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:35.622387 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:35.622698 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.623197 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.624321 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.624707 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.624837 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.624890 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:35.625173 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 02:38:35.625245 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 02:38:35.625371 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.621603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:44483 (local address 127.7.194.193:56623)
0619 02:38:35.621994 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:35.622009 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:35.622140 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:35.622408 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:35.622411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:35.622426 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:35.622571 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.622582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.623309 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.623315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.624186 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.624195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.624833 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.624837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.624932 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.625005 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 02:38:35.625095 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:38:35.625207 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":94,"thread_start_us":83,"threads_started":1}
I20260619 02:38:35.625355 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.621560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:45679 (local address 127.7.194.193:43597)
0619 02:38:35.621904 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:35.621916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:35.622062 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:35.622362 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:35.622365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:35.622381 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:35.622541 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.622547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.623292 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.623296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.624198 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:35.624207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.624868 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.624872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:35.624990 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.625008 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:38:35.625094 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:38:35.625201 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":67,"thread_start_us":75,"threads_started":1}
I20260619 02:38:35.626098 27052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af0468f4203476e92bd50f5838267e3" candidate_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" is_pre_election: true
I20260619 02:38:35.626187 26921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af0468f4203476e92bd50f5838267e3" candidate_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6adb26eef14696becee1ae568558fc" is_pre_election: true
I20260619 02:38:35.626446 27052 raft_consensus.cc:2454] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff25f75bac6f46a8a89bd4e7f1d9de51 in term 0.
I20260619 02:38:35.626488 26921 raft_consensus.cc:2454] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff25f75bac6f46a8a89bd4e7f1d9de51 in term 0.
I20260619 02:38:35.626991 26726 leader_election.cc:302] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04f38b859b9a4c2298cfd5cb801c1f5c, 1c6adb26eef14696becee1ae568558fc, ff25f75bac6f46a8a89bd4e7f1d9de51; no voters:
I20260619 02:38:35.627277 27123 raft_consensus.cc:2781] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:38:35.627354 27123 raft_consensus.cc:482] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:38:35.627384 27123 raft_consensus.cc:3037] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:35.628504 27123 raft_consensus.cc:504] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } }
I20260619 02:38:35.628937 27123 leader_election.cc:288] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [CANDIDATE]: Term 1 election: Requested vote from peers 1c6adb26eef14696becee1ae568558fc (127.7.194.194:45679), 04f38b859b9a4c2298cfd5cb801c1f5c (127.7.194.195:44483)
I20260619 02:38:35.629329 26921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af0468f4203476e92bd50f5838267e3" candidate_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6adb26eef14696becee1ae568558fc"
I20260619 02:38:35.629366 27052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af0468f4203476e92bd50f5838267e3" candidate_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c"
I20260619 02:38:35.629472 27052 raft_consensus.cc:3037] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:35.629472 26921 raft_consensus.cc:3037] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:35.630826 26921 raft_consensus.cc:2454] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff25f75bac6f46a8a89bd4e7f1d9de51 in term 1.
I20260619 02:38:35.630826 27052 raft_consensus.cc:2454] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff25f75bac6f46a8a89bd4e7f1d9de51 in term 1.
I20260619 02:38:35.631142 26726 leader_election.cc:302] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f38b859b9a4c2298cfd5cb801c1f5c, ff25f75bac6f46a8a89bd4e7f1d9de51; no voters:
I20260619 02:38:35.631353 27123 raft_consensus.cc:2781] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:35.631593 27123 raft_consensus.cc:686] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [term 1 LEADER]: Becoming Leader. State: Replica: ff25f75bac6f46a8a89bd4e7f1d9de51, State: Running, Role: LEADER
I20260619 02:38:35.631953 27123 consensus_queue.cc:237] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } }
I20260619 02:38:35.635150 26647 catalog_manager.cc:5495] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff25f75bac6f46a8a89bd4e7f1d9de51 (127.7.194.193). New cstate: current_term: 1 leader_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff25f75bac6f46a8a89bd4e7f1d9de51" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41479 } health_report { overall_health: HEALTHY } } }
I20260619 02:38:35.685745 27114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.682077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.7.194.193:41479)
0619 02:38:35.682243 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:35.682247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:35.682262 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:38:35.682351 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:35.682353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:35.682412 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:35.682505 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:38:35.682735 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.683228 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.684109 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:35.684411 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:35.684521 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:35.684837 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:38:35.685262 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:38:35.685373 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:35.685486 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 02:38:35.685554 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 02:38:35.692021 26921 raft_consensus.cc:1261] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [term 1 FOLLOWER]: Refusing update from remote peer ff25f75bac6f46a8a89bd4e7f1d9de51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:35.692022 27052 raft_consensus.cc:1261] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [term 1 FOLLOWER]: Refusing update from remote peer ff25f75bac6f46a8a89bd4e7f1d9de51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:35.692792 27126 consensus_queue.cc:1035] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c6adb26eef14696becee1ae568558fc" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:35.692997 27123 consensus_queue.cc:1035] T 2af0468f4203476e92bd50f5838267e3 P ff25f75bac6f46a8a89bd4e7f1d9de51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f38b859b9a4c2298cfd5cb801c1f5c" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:35.701299 27130 mvcc.cc:205] Tried to move back new op lower bound from 7298403187466604544 to 7298403187231043584. Current Snapshot: MvccSnapshot[applied={T|T < 7298403187466604544}]
I20260619 02:38:35.704897 27132 mvcc.cc:205] Tried to move back new op lower bound from 7298403187466604544 to 7298403187231043584. Current Snapshot: MvccSnapshot[applied={T|T < 7298403187466604544}]
I20260619 02:38:35.710986 26647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:35.711169 26647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:35.713668 26647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1863f6e63f15418c817db90342b07979: Sending DeleteTablet for 3 replicas of tablet 2af0468f4203476e92bd50f5838267e3
I20260619 02:38:35.714449 26901 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af0468f4203476e92bd50f5838267e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:35 UTC) from {username='slave'} at 127.0.0.1:39520
I20260619 02:38:35.714557 27032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af0468f4203476e92bd50f5838267e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:35 UTC) from {username='slave'} at 127.0.0.1:44860
I20260619 02:38:35.714977 27139 tablet_replica.cc:330] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: stopping tablet replica
I20260619 02:38:35.715219 26770 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af0468f4203476e92bd50f5838267e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:35 UTC) from {username='slave'} at 127.0.0.1:60326
I20260619 02:38:35.715457 27139 raft_consensus.cc:2229] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:35.715559 27140 tablet_replica.cc:330] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: stopping tablet replica
I20260619 02:38:35.715790 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 26708
I20260619 02:38:35.715864 27140 raft_consensus.cc:2229] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:35.716168 27140 raft_consensus.cc:2258] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:35.717964 27140 ts_tablet_manager.cc:1905] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:35.718520 27139 raft_consensus.cc:2258] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:35.720140 27139 ts_tablet_manager.cc:1905] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:35.720441 27140 ts_tablet_manager.cc:1918] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:35.720551 27140 log.cc:1196] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/2af0468f4203476e92bd50f5838267e3
I20260619 02:38:35.720858 27140 ts_tablet_manager.cc:1939] T 2af0468f4203476e92bd50f5838267e3 P 04f38b859b9a4c2298cfd5cb801c1f5c: Deleting consensus metadata
I20260619 02:38:35.722066 26635 catalog_manager.cc:4841] TS 04f38b859b9a4c2298cfd5cb801c1f5c (127.7.194.195:44483): tablet 2af0468f4203476e92bd50f5838267e3 (table test-table [id=e250e219c5384192a315a2ff573ee959]) successfully deleted
W20260619 02:38:35.723282 26635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:41479 (error 108)
I20260619 02:38:35.723523 27139 ts_tablet_manager.cc:1918] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:35.723627 27139 log.cc:1196] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/2af0468f4203476e92bd50f5838267e3
I20260619 02:38:35.723958 27139 ts_tablet_manager.cc:1939] T 2af0468f4203476e92bd50f5838267e3 P 1c6adb26eef14696becee1ae568558fc: Deleting consensus metadata
I20260619 02:38:35.724139 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 26839
I20260619 02:38:35.724418 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.724101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41479 (local address 127.0.0.1:60332)
0619 02:38:35.724243 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:35.724319 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 02:38:35.724741 26635 catalog_manager.cc:4584] TS ff25f75bac6f46a8a89bd4e7f1d9de51 (127.7.194.193:41479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2af0468f4203476e92bd50f5838267e3: Network error: Client connection negotiation failed: client connection to 127.7.194.193:41479: connect: Connection refused (error 111)
I20260619 02:38:35.730230 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 26970
I20260619 02:38:35.731453 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:35.730368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:45679 (local address 127.0.0.1:39526)
0619 02:38:35.731327 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:35.731378 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.194:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":853}
I20260619 02:38:35.735394 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 26617
2026-06-19T02:38:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1412 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:35 dist-test-slave-9tkz krb5kdc[27145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:35 dist-test-slave-9tkz krb5kdc[27145](info): set up 2 sockets
Jun 19 02:38:35 dist-test-slave-9tkz krb5kdc[27145](info): commencing operation
krb5kdc: starting...
W20260619 02:38:36.769757 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:36 dist-test-slave-9tkz krb5kdc[27145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:38:36.902877 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:46333
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:36081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:46333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:37.017760 27158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:37.018030 27158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:37.018072 27158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:37.022233 27158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:37.022295 27158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:37.022317 27158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:37.022334 27158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:37.022392 27158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:37.027647 27158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:36081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:46333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:37.028824 27158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:37.029914 27158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:37.038460 27163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:37.038455 27164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:37.038455 27166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:37.038894 27158 server_base.cc:1034] running on GCE node
I20260619 02:38:37.039426 27158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:37.040531 27158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:37.041715 27158 hybrid_clock.cc:648] HybridClock initialized: now 1781836717041702 us; error 26 us; skew 500 ppm
Jun 19 02:38:37 dist-test-slave-9tkz krb5kdc[27145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836717, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:37.044517 27158 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:38:37.045378 27158 webserver.cc:460] Webserver started at http://127.7.194.254:41735/ using document root <none> and password file <none>
I20260619 02:38:37.046005 27158 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:37.046097 27158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:37.046365 27158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:37.048413 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "262e37ee72cf4b93a419bd22126ef1e9"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "4e476fe2a88cd84b5bd8f5c7a2e98b8a"
server_key_iv: "2fc36d74f8fe2f8c15ca7c68f138d915"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.048972 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "262e37ee72cf4b93a419bd22126ef1e9"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "4e476fe2a88cd84b5bd8f5c7a2e98b8a"
server_key_iv: "2fc36d74f8fe2f8c15ca7c68f138d915"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.052758 27158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:38:37.055333 27173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.056860 27158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:38:37.056957 27158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "262e37ee72cf4b93a419bd22126ef1e9"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "4e476fe2a88cd84b5bd8f5c7a2e98b8a"
server_key_iv: "2fc36d74f8fe2f8c15ca7c68f138d915"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.057049 27158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:37.087527 27158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:37.090764 27158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:37.090926 27158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:37.098016 27158 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:46333
I20260619 02:38:37.098044 27225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:46333 every 8 connection(s)
I20260619 02:38:37.098966 27158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:37.099805 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 27158
I20260619 02:38:37.099942 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:37.100226 7947 external_mini_cluster.cc:1428] Setting key 646d45c882a6f26171f2dfed88c3a1a0
I20260619 02:38:37.102209 27226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:37.108879 27226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9: Bootstrap starting.
Jun 19 02:38:37 dist-test-slave-9tkz krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:37.111500 27226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:37.112270 27226 log.cc:824] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:37.114598 27226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9: No bootstrap required, opened a new log
I20260619 02:38:37.115800 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.101700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.7.194.254:46333)
0619 02:38:37.102114 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:37.102135 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:37.102177 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:38:37.102993 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:37.103015 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:37.103421 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:37.103809 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:37.104630 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.106136 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.107029 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.107730 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.107899 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.110739 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:37.110744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:37.110756 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:37.110782 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:37.112614 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.113063 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.113068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.113093 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.113257 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.113629 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.113632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.113634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.114145 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:37.114337 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:37.114565 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 02:38:37.114867 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260619 02:38:37.117904 27226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262e37ee72cf4b93a419bd22126ef1e9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46333 } }
I20260619 02:38:37.118152 27226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:37.118235 27226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 262e37ee72cf4b93a419bd22126ef1e9, State: Initialized, Role: FOLLOWER
I20260619 02:38:37.118634 27226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262e37ee72cf4b93a419bd22126ef1e9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46333 } }
I20260619 02:38:37.118781 27226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:37.118872 27226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:37.118969 27226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:37.119848 27226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262e37ee72cf4b93a419bd22126ef1e9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46333 } }
I20260619 02:38:37.120185 27226 leader_election.cc:302] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 262e37ee72cf4b93a419bd22126ef1e9; no voters:
I20260619 02:38:37.120509 27226 leader_election.cc:288] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:37.120643 27231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:37.120908 27231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [term 1 LEADER]: Becoming Leader. State: Replica: 262e37ee72cf4b93a419bd22126ef1e9, State: Running, Role: LEADER
I20260619 02:38:37.121243 27231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262e37ee72cf4b93a419bd22126ef1e9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46333 } }
I20260619 02:38:37.121829 27226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:37.123126 27232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "262e37ee72cf4b93a419bd22126ef1e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262e37ee72cf4b93a419bd22126ef1e9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46333 } } }
I20260619 02:38:37.123107 27233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 262e37ee72cf4b93a419bd22126ef1e9. Latest consensus state: current_term: 1 leader_uuid: "262e37ee72cf4b93a419bd22126ef1e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262e37ee72cf4b93a419bd22126ef1e9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46333 } } }
I20260619 02:38:37.123275 27232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:37.123275 27233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:37.123703 27241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:37.126945 27241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:37.132040 27241 catalog_manager.cc:1347] Generated new cluster ID: aa821026d7b0437fa40eb9a26c252c81
I20260619 02:38:37.132103 27241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:37.141889 27241 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:37.142867 27241 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:37.157863 27241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9: Generated new TSK 0
I20260619 02:38:37.158803 27241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
Entry for principal kudu/127.7.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab.
I20260619 02:38:37.194758 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46333
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:36081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:37.306979 27252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:37.307261 27252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:37.307304 27252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:37.311650 27252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:37.311714 27252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:37.311808 27252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:38:37.317159 27252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:36081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.193.keytab
--principal=kudu/127.7.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:37.318277 27252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:37.319339 27252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:37.329082 27258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:37.329121 27257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:37.329092 27260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:37.329334 27252 server_base.cc:1034] running on GCE node
I20260619 02:38:37.329818 27252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:37.330426 27252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:37.331607 27252 hybrid_clock.cc:648] HybridClock initialized: now 1781836717331577 us; error 51 us; skew 500 ppm
Jun 19 02:38:37 dist-test-slave-9tkz krb5kdc[27145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836717, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:37.334329 27252 init.cc:377] Logged in from keytab as kudu/127.7.194.193@KRBTEST.COM (short username kudu)
I20260619 02:38:37.335189 27252 webserver.cc:460] Webserver started at http://127.7.194.193:46065/ using document root <none> and password file <none>
I20260619 02:38:37.335767 27252 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:37.335858 27252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:37.336159 27252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:37.338269 27252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "6900142e2d394442b8e47ef3b1ba646d"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "a18bcb8777d6df9d464a90528bbef32b"
server_key_iv: "3683bf06153584be59e09a5d225fbf7f"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.338826 27252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "6900142e2d394442b8e47ef3b1ba646d"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "a18bcb8777d6df9d464a90528bbef32b"
server_key_iv: "3683bf06153584be59e09a5d225fbf7f"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.342689 27252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:38:37.345177 27267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.346490 27252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:37.346633 27252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "6900142e2d394442b8e47ef3b1ba646d"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "a18bcb8777d6df9d464a90528bbef32b"
server_key_iv: "3683bf06153584be59e09a5d225fbf7f"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.346745 27252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:37.362684 27252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:37.365774 27252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:37.365965 27252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:37.366539 27252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:37.367341 27252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:37.367410 27252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.367482 27252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:37.367532 27252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.377349 27252 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:41933
I20260619 02:38:37.377367 27380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:41933 every 8 connection(s)
I20260619 02:38:37.378300 27252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:38:37.381166 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 27252
I20260619 02:38:37.381299 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:38:37.381699 7947 external_mini_cluster.cc:1428] Setting key 8ba1e1ad5dfcf5b76c60ba78a194d901
Jun 19 02:38:37 dist-test-slave-9tkz krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836717, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.193@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.194@KRBTEST.COM; defaulting to no policy
I20260619 02:38:37.392864 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.380077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:34809 (local address 127.7.194.254:46333)
0619 02:38:37.380206 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:37.380210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:37.381016 (+ 806us) server_negotiation.cc:407] Connection header received
0619 02:38:37.382723 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:37.382726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:37.382781 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:37.382885 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:37.384484 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.385118 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.386362 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.386601 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.386723 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.389184 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:37.389188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:37.389191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:37.389220 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:37.390972 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.391489 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.391493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.391495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.391603 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.391908 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.391911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.391912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.392092 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:37.392170 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:37.392621 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 02:38:37.392736 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.194@KRBTEST.COM" created.
I20260619 02:38:37.393807 27383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.380397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:46333 (local address 127.7.194.193:34809)
0619 02:38:37.380842 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:37.380885 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:37.382500 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:37.383071 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:37.383078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:37.383565 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:37.384309 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.384333 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.385254 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.385262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.386233 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.386242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.386763 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.386767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.386934 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.387545 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:37.387562 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:37.388998 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:37.391078 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:37.391083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:37.391094 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:37.391408 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:37.391696 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:37.391698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:37.391700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:37.391827 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:37.392179 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:37.392184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:37.392411 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 02:38:37.392566 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 02:38:37.392826 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":118,"threads_started":1}
I20260619 02:38:37.395098 27381 heartbeater.cc:347] Connected to a master server at 127.7.194.254:46333
I20260619 02:38:37.395375 27381 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:37.395903 27381 heartbeater.cc:511] Master 127.7.194.254:46333 requested a full tablet report, sending...
I20260619 02:38:37.397480 27190 ts_manager.cc:194] Registered new tserver with Master: 6900142e2d394442b8e47ef3b1ba646d (127.7.194.193:41933)
I20260619 02:38:37.398663 27190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.193@KRBTEST.COM'} at 127.7.194.193:34809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
Entry for principal kudu/127.7.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab.
I20260619 02:38:37.408649 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46333
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:36081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:37.521860 27386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:37.522135 27386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:37.522178 27386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:37.526556 27386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:37.526620 27386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:37.526682 27386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:38:37.531945 27386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:36081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.194.keytab
--principal=kudu/127.7.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:37.532955 27386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:37.534093 27386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:37.543721 27391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:37.543908 27386 server_base.cc:1034] running on GCE node
W20260619 02:38:37.543706 27394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:37.543706 27392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:37.544641 27386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:37.545264 27386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:37.546466 27386 hybrid_clock.cc:648] HybridClock initialized: now 1781836717546449 us; error 36 us; skew 500 ppm
Jun 19 02:38:37 dist-test-slave-9tkz krb5kdc[27145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836717, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:37.549327 27386 init.cc:377] Logged in from keytab as kudu/127.7.194.194@KRBTEST.COM (short username kudu)
I20260619 02:38:37.550333 27386 webserver.cc:460] Webserver started at http://127.7.194.194:39425/ using document root <none> and password file <none>
I20260619 02:38:37.550956 27386 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:37.551047 27386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:37.551281 27386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:37.553316 27386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "0c409573b63a4823816821321eafd402"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "35d6affc1786ba70ca31e59d8c22c0a7"
server_key_iv: "c257705a9f443d40112e0f99e35a18b0"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.553939 27386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "0c409573b63a4823816821321eafd402"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "35d6affc1786ba70ca31e59d8c22c0a7"
server_key_iv: "c257705a9f443d40112e0f99e35a18b0"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.557629 27386 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:38:37.560007 27401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.561271 27386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:37.561400 27386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "0c409573b63a4823816821321eafd402"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "35d6affc1786ba70ca31e59d8c22c0a7"
server_key_iv: "c257705a9f443d40112e0f99e35a18b0"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.561508 27386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:37.574015 27386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:37.576797 27386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:37.576999 27386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:37.577643 27386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:37.578434 27386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:37.578506 27386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.578573 27386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:37.578625 27386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.587613 27386 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:43347
I20260619 02:38:37.587622 27514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:43347 every 8 connection(s)
I20260619 02:38:37.588498 27386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:38:37.595021 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 27386
I20260619 02:38:37.595158 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:38:37.595479 7947 external_mini_cluster.cc:1428] Setting key 1ffc85d63dac905ae01bcfb7a608ea8d
Jun 19 02:38:37 dist-test-slave-9tkz krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836717, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.194@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:37.603236 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.590324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:35413 (local address 127.7.194.254:46333)
0619 02:38:37.590456 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:37.590460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:37.591264 (+ 804us) server_negotiation.cc:407] Connection header received
0619 02:38:37.592572 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:37.592575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:37.592636 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:37.592760 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:37.594013 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.594562 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.596169 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.596405 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.596517 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.599242 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:37.599249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:37.599253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:37.599287 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:37.600768 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.601396 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.601398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.601400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.601471 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.601898 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.601901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.601902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.602183 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:37.602317 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:37.602831 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 02:38:37.602975 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:38:37.603906 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.590707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:46333 (local address 127.7.194.194:35413)
0619 02:38:37.591126 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:37.591166 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:37.592366 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:37.592929 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:37.592936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:37.593311 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:37.593872 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.593886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.594686 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.594691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.596036 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.596045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.596554 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.596558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.596731 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.597333 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:37.597354 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:37.599060 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:37.600885 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:37.600894 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:37.600913 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:37.601297 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:37.601613 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:37.601616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:37.601618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:37.601805 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:37.602315 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:37.602320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:37.602559 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 02:38:37.602722 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 02:38:37.602984 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":113,"threads_started":1}
I20260619 02:38:37.605108 27515 heartbeater.cc:347] Connected to a master server at 127.7.194.254:46333
I20260619 02:38:37.605365 27515 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.194.195@KRBTEST.COM; defaulting to no policy
I20260619 02:38:37.605929 27515 heartbeater.cc:511] Master 127.7.194.254:46333 requested a full tablet report, sending...
I20260619 02:38:37.606949 27190 ts_manager.cc:194] Registered new tserver with Master: 0c409573b63a4823816821321eafd402 (127.7.194.194:43347)
I20260619 02:38:37.607488 27190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.194@KRBTEST.COM'} at 127.7.194.194:35413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
Entry for principal kudu/127.7.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab.
I20260619 02:38:37.622738 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46333
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:36081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:37.735726 27520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:37.736006 27520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:37.736084 27520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:37.740299 27520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:37.740355 27520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:37.740450 27520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:38:37.745725 27520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:36081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.195.keytab
--principal=kudu/127.7.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:46333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:37.746807 27520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:37.747835 27520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:37.757505 27526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:37.757652 27520 server_base.cc:1034] running on GCE node
W20260619 02:38:37.757563 27528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:37.757575 27525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:37.758291 27520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:37.758952 27520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:37.760282 27520 hybrid_clock.cc:648] HybridClock initialized: now 1781836717760243 us; error 46 us; skew 500 ppm
Jun 19 02:38:37 dist-test-slave-9tkz krb5kdc[27145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836717, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:37.763231 27520 init.cc:377] Logged in from keytab as kudu/127.7.194.195@KRBTEST.COM (short username kudu)
I20260619 02:38:37.764076 27520 webserver.cc:460] Webserver started at http://127.7.194.195:39075/ using document root <none> and password file <none>
I20260619 02:38:37.764678 27520 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:37.764763 27520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:37.764967 27520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:37.767059 27520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "9a7c3f6796c143f3882dc72671fbf0a9"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "57c3e8cc5226c5dec27531d283e7a3ce"
server_key_iv: "5d67021fad6d9700c28113ac1e677631"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.767604 27520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "9a7c3f6796c143f3882dc72671fbf0a9"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "57c3e8cc5226c5dec27531d283e7a3ce"
server_key_iv: "5d67021fad6d9700c28113ac1e677631"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.771337 27520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:38:37.773720 27535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.774994 27520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:37.775158 27520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "9a7c3f6796c143f3882dc72671fbf0a9"
format_stamp: "Formatted at 2026-06-19 02:38:37 on dist-test-slave-9tkz"
server_key: "57c3e8cc5226c5dec27531d283e7a3ce"
server_key_iv: "5d67021fad6d9700c28113ac1e677631"
server_key_version: "encryptionkey@0"
I20260619 02:38:37.775254 27520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:37.800873 27520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:37.803984 27520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:37.804205 27520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:37.804804 27520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:37.805663 27520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:37.805711 27520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.805781 27520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:37.805816 27520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:37.814771 27520 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:32865
I20260619 02:38:37.814813 27648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:32865 every 8 connection(s)
I20260619 02:38:37.815721 27520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:38:37.819530 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 27520
I20260619 02:38:37.819641 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:38:37.819964 7947 external_mini_cluster.cc:1428] Setting key 7de9c2e6780ceff4e85f1bf8a9cd89e4
Jun 19 02:38:37 dist-test-slave-9tkz krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836717, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.195@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:37.829293 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.817637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:51769 (local address 127.7.194.254:46333)
0619 02:38:37.817775 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:37.817778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:37.818621 (+ 843us) server_negotiation.cc:407] Connection header received
0619 02:38:37.820014 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:37.820017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:37.820079 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:37.820166 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:37.821585 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.822214 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.823479 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.823710 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.823820 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.826274 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:37.826276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:37.826278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:37.826301 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:37.827528 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.827978 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.827981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.827982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.828053 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.828341 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.828343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.828344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.828523 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:37.828599 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:37.829073 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 02:38:37.829174 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:38:37.830149 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.818016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:46333 (local address 127.7.194.195:51769)
0619 02:38:37.818458 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:37.818501 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:37.819795 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:37.820350 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:37.820357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:37.820829 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:37.821419 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.821440 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.822334 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.822338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.823363 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.823372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.823858 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.823862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.824033 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.824652 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:38:37.824668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:37.826127 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:37.827641 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:37.827645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:37.827657 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:37.827898 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:37.828131 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:38:37.828133 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:38:37.828134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:38:37.828257 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:38:37.828638 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:37.828643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:37.828871 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 02:38:37.829033 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 02:38:37.829281 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260619 02:38:37.831351 27649 heartbeater.cc:347] Connected to a master server at 127.7.194.254:46333
I20260619 02:38:37.831564 27649 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:37.832011 27649 heartbeater.cc:511] Master 127.7.194.254:46333 requested a full tablet report, sending...
I20260619 02:38:37.833117 27189 ts_manager.cc:194] Registered new tserver with Master: 9a7c3f6796c143f3882dc72671fbf0a9 (127.7.194.195:32865)
I20260619 02:38:37.833912 27189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.194.195@KRBTEST.COM'} at 127.7.194.195:51769
I20260619 02:38:37.833952 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:38:37.843081 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.835954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.7.194.254:46333)
0619 02:38:37.836104 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:37.836109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:37.836190 (+ 81us) server_negotiation.cc:407] Connection header received
0619 02:38:37.836288 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:37.836290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:37.836324 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:37.836388 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:37.837027 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.837500 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.838370 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.838575 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.838667 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.839991 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:37.839994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:37.839996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:37.840017 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:37.841465 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.841932 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.841936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.841937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.842017 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:37.842277 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:37.842280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:37.842281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:37.842482 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:37.842597 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:37.842761 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 02:38:37.842862 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:38:37.846112 27189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:38:37.849030 27189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:38:37.864041 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.858067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.7.194.194:43347)
0619 02:38:37.858488 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:37.858504 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:37.858966 (+ 462us) server_negotiation.cc:407] Connection header received
0619 02:38:37.859061 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:37.859067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:37.859280 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:37.859447 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:37.859959 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.861182 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.862702 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.863344 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.863473 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.863629 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:37.863797 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:38:37.863886 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":145,"threads_started":1}
I20260619 02:38:37.864032 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.857618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:43347 (local address 127.0.0.1:55304)
0619 02:38:37.858597 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:37.858649 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:37.858802 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:37.859647 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:37.859649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:37.859691 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:37.859811 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.859818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.861406 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.861410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.862566 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.862577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.863473 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.863476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.863590 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.863625 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:38:37.863757 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:38:37.863881 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":78,"spinlock_wait_cycles":1792,"thread_start_us":74,"threads_started":1}
I20260619 02:38:37.864293 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.857735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:32865 (local address 127.0.0.1:39668)
0619 02:38:37.858856 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:37.858872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:37.858960 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:37.859448 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:37.859454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:37.859477 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:37.859614 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.859623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.860842 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.860849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.862375 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.862384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.863704 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.863708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.863814 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.863839 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:38:37.863914 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:38:37.863990 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":18,"thread_start_us":105,"threads_started":1}
I20260619 02:38:37.864472 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.857932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.7.194.195:32865)
0619 02:38:37.858709 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:37.858724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:37.859074 (+ 350us) server_negotiation.cc:407] Connection header received
0619 02:38:37.859147 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:37.859152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:37.859295 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:37.859447 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:37.859752 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.860704 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.862519 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.863343 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.863460 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.863631 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:37.864114 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 02:38:37.864232 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":189,"threads_started":1}
I20260619 02:38:37.864534 27665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.858394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41933 (local address 127.0.0.1:42332)
0619 02:38:37.858939 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:37.858953 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:37.859036 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:37.859714 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:37.859716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:37.859743 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:37.859818 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.859824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.861323 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.861326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.862784 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:37.862793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.864035 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.864041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:37.864133 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.864156 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:38:37.864231 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:38:37.864290 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":299,"spinlock_wait_cycles":4224,"thread_start_us":78,"threads_started":1}
I20260619 02:38:37.864674 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:37.858551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.7.194.193:41933)
0619 02:38:37.859263 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:37.859276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:37.859298 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:38:37.859374 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:37.859381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:37.859575 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:37.859730 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:37.859959 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.861182 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.862908 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:37.863717 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:37.863824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:37.863960 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:37.864362 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 02:38:37.864467 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":129,"threads_started":1}
I20260619 02:38:37.867280 27583 tablet_service.cc:1467] Processing CreateTablet for tablet 249122c944aa464ab8435154418adbb8 (DEFAULT_TABLE table=test-table [id=bae5c82d8ce4434c87cab3dd2bea1b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:37.867424 27449 tablet_service.cc:1467] Processing CreateTablet for tablet 249122c944aa464ab8435154418adbb8 (DEFAULT_TABLE table=test-table [id=bae5c82d8ce4434c87cab3dd2bea1b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:37.867422 27315 tablet_service.cc:1467] Processing CreateTablet for tablet 249122c944aa464ab8435154418adbb8 (DEFAULT_TABLE table=test-table [id=bae5c82d8ce4434c87cab3dd2bea1b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:37.868197 27583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249122c944aa464ab8435154418adbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:37.868392 27449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249122c944aa464ab8435154418adbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:37.868392 27315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249122c944aa464ab8435154418adbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:37.874985 27666 tablet_bootstrap.cc:492] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402: Bootstrap starting.
I20260619 02:38:37.876389 27667 tablet_bootstrap.cc:492] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: Bootstrap starting.
I20260619 02:38:37.876696 27668 tablet_bootstrap.cc:492] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d: Bootstrap starting.
I20260619 02:38:37.877743 27666 tablet_bootstrap.cc:654] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:37.878821 27668 tablet_bootstrap.cc:654] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:37.878821 27667 tablet_bootstrap.cc:654] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:37.879027 27666 log.cc:824] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:37.880280 27668 log.cc:824] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:37.880280 27667 log.cc:824] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:37.881251 27666 tablet_bootstrap.cc:492] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402: No bootstrap required, opened a new log
I20260619 02:38:37.881446 27666 ts_tablet_manager.cc:1397] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 02:38:37.882177 27668 tablet_bootstrap.cc:492] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d: No bootstrap required, opened a new log
I20260619 02:38:37.882339 27667 tablet_bootstrap.cc:492] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: No bootstrap required, opened a new log
I20260619 02:38:37.882422 27668 ts_tablet_manager.cc:1397] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:38:37.882521 27667 ts_tablet_manager.cc:1397] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:38:37.885463 27666 raft_consensus.cc:348] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } }
I20260619 02:38:37.885877 27666 raft_consensus.cc:374] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:37.885955 27666 raft_consensus.cc:729] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c409573b63a4823816821321eafd402, State: Initialized, Role: FOLLOWER
I20260619 02:38:37.885797 27668 raft_consensus.cc:348] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } }
I20260619 02:38:37.886068 27668 raft_consensus.cc:374] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:37.886138 27668 raft_consensus.cc:729] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6900142e2d394442b8e47ef3b1ba646d, State: Initialized, Role: FOLLOWER
I20260619 02:38:37.886291 27667 raft_consensus.cc:348] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } }
I20260619 02:38:37.886545 27667 raft_consensus.cc:374] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:37.886546 27666 consensus_queue.cc:260] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } }
I20260619 02:38:37.886636 27667 raft_consensus.cc:729] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7c3f6796c143f3882dc72671fbf0a9, State: Initialized, Role: FOLLOWER
I20260619 02:38:37.886623 27668 consensus_queue.cc:260] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } }
I20260619 02:38:37.887022 27667 consensus_queue.cc:260] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } }
I20260619 02:38:37.887395 27381 heartbeater.cc:503] Master 127.7.194.254:46333 was elected leader, sending a full tablet report...
I20260619 02:38:37.887763 27668 ts_tablet_manager.cc:1428] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:38:37.887987 27667 ts_tablet_manager.cc:1428] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:38:37.888505 27649 heartbeater.cc:503] Master 127.7.194.254:46333 was elected leader, sending a full tablet report...
I20260619 02:38:37.889658 27666 ts_tablet_manager.cc:1428] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 02:38:37.890007 27515 heartbeater.cc:503] Master 127.7.194.254:46333 was elected leader, sending a full tablet report...
I20260619 02:38:38.057636 27673 raft_consensus.cc:482] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:38.057899 27673 raft_consensus.cc:504] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } }
I20260619 02:38:38.059079 27673 leader_election.cc:288] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c409573b63a4823816821321eafd402 (127.7.194.194:43347), 9a7c3f6796c143f3882dc72671fbf0a9 (127.7.194.195:32865)
I20260619 02:38:38.064843 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.059539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:53585 (local address 127.7.194.194:43347)
0619 02:38:38.059791 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:38.059794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:38.060055 (+ 261us) server_negotiation.cc:407] Connection header received
0619 02:38:38.060111 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:38.060113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:38.060161 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:38.060256 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:38.060715 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.061465 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.063433 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.064098 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.064241 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.064300 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:38.064607 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 02:38:38.064702 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260619 02:38:38.064843 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.059548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:53887 (local address 127.7.194.195:32865)
0619 02:38:38.059679 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:38.059685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:38.059927 (+ 242us) server_negotiation.cc:407] Connection header received
0619 02:38:38.060101 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:38.060104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:38.060158 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:38.060254 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:38.060661 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.061465 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.063433 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.064098 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.064241 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.064300 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:38.064623 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 02:38:38.064702 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:38:38.064869 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.059414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:43347 (local address 127.7.194.193:53585)
0619 02:38:38.059941 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:38.059953 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:38.060038 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:38.060279 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:38.060282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:38.060322 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:38.060533 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.060542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.061638 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.061642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.063269 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.063281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.064242 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.064246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.064441 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.064490 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 02:38:38.064573 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:38:38.064657 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":312,"spinlock_wait_cycles":27776,"thread_start_us":95,"threads_started":1}
I20260619 02:38:38.064872 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.059414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:32865 (local address 127.7.194.193:53887)
0619 02:38:38.059819 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:38.059837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:38.059975 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:38.060279 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:38.060282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:38.060323 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:38.060540 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.060545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.061638 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.061642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.063269 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.063281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.064242 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.064246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.064450 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.064494 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:38:38.064573 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:38:38.064657 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":313,"spinlock_wait_cycles":42368,"thread_start_us":92,"threads_started":1}
I20260619 02:38:38.065593 27469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249122c944aa464ab8435154418adbb8" candidate_uuid: "6900142e2d394442b8e47ef3b1ba646d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c409573b63a4823816821321eafd402" is_pre_election: true
I20260619 02:38:38.065974 27469 raft_consensus.cc:2454] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6900142e2d394442b8e47ef3b1ba646d in term 0.
I20260619 02:38:38.066201 27603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249122c944aa464ab8435154418adbb8" candidate_uuid: "6900142e2d394442b8e47ef3b1ba646d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" is_pre_election: true
I20260619 02:38:38.066533 27603 raft_consensus.cc:2454] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6900142e2d394442b8e47ef3b1ba646d in term 0.
I20260619 02:38:38.066542 27270 leader_election.cc:302] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c409573b63a4823816821321eafd402, 6900142e2d394442b8e47ef3b1ba646d; no voters:
W20260619 02:38:38.066661 27650 tablet.cc:2369] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:38:38.066826 27673 raft_consensus.cc:2781] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:38:38.066946 27673 raft_consensus.cc:482] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:38:38.067008 27673 raft_consensus.cc:3037] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:38.068068 27673 raft_consensus.cc:504] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } }
I20260619 02:38:38.068445 27673 leader_election.cc:288] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [CANDIDATE]: Term 1 election: Requested vote from peers 0c409573b63a4823816821321eafd402 (127.7.194.194:43347), 9a7c3f6796c143f3882dc72671fbf0a9 (127.7.194.195:32865)
I20260619 02:38:38.068753 27469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249122c944aa464ab8435154418adbb8" candidate_uuid: "6900142e2d394442b8e47ef3b1ba646d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c409573b63a4823816821321eafd402"
I20260619 02:38:38.068789 27603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249122c944aa464ab8435154418adbb8" candidate_uuid: "6900142e2d394442b8e47ef3b1ba646d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7c3f6796c143f3882dc72671fbf0a9"
I20260619 02:38:38.068892 27469 raft_consensus.cc:3037] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:38.068900 27603 raft_consensus.cc:3037] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:38.070055 27603 raft_consensus.cc:2454] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6900142e2d394442b8e47ef3b1ba646d in term 1.
I20260619 02:38:38.070130 27469 raft_consensus.cc:2454] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6900142e2d394442b8e47ef3b1ba646d in term 1.
I20260619 02:38:38.070369 27268 leader_election.cc:302] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6900142e2d394442b8e47ef3b1ba646d, 9a7c3f6796c143f3882dc72671fbf0a9; no voters:
I20260619 02:38:38.070546 27673 raft_consensus.cc:2781] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:38.070808 27673 raft_consensus.cc:686] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [term 1 LEADER]: Becoming Leader. State: Replica: 6900142e2d394442b8e47ef3b1ba646d, State: Running, Role: LEADER
I20260619 02:38:38.071130 27673 consensus_queue.cc:237] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } }
I20260619 02:38:38.074143 27189 catalog_manager.cc:5495] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6900142e2d394442b8e47ef3b1ba646d (127.7.194.193). New cstate: current_term: 1 leader_uuid: "6900142e2d394442b8e47ef3b1ba646d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6900142e2d394442b8e47ef3b1ba646d" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 41933 } health_report { overall_health: HEALTHY } } }
W20260619 02:38:38.089686 27516 tablet.cc:2369] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:38:38.105634 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.101343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.7.194.193:41933)
0619 02:38:38.101499 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:38.101504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:38.101520 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:38:38.101698 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:38.101701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:38.101769 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:38.101877 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:38:38.102130 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.102920 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.103827 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.104160 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.104256 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.104652 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:38:38.105141 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:38:38.105249 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:38.105412 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 02:38:38.105472 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:38:38.112093 27469 raft_consensus.cc:1261] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [term 1 FOLLOWER]: Refusing update from remote peer 6900142e2d394442b8e47ef3b1ba646d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:38.112107 27603 raft_consensus.cc:1261] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [term 1 FOLLOWER]: Refusing update from remote peer 6900142e2d394442b8e47ef3b1ba646d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:38.112854 27677 consensus_queue.cc:1035] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a7c3f6796c143f3882dc72671fbf0a9" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:38.113030 27673 consensus_queue.cc:1035] T 249122c944aa464ab8435154418adbb8 P 6900142e2d394442b8e47ef3b1ba646d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c409573b63a4823816821321eafd402" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 43347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:38.122584 27681 mvcc.cc:205] Tried to move back new op lower bound from 7298403197379698688 to 7298403197221797888. Current Snapshot: MvccSnapshot[applied={T|T < 7298403197379698688}]
I20260619 02:38:38.122735 27682 mvcc.cc:205] Tried to move back new op lower bound from 7298403197379698688 to 7298403197221797888. Current Snapshot: MvccSnapshot[applied={T|T < 7298403197379698688}]
I20260619 02:38:38.128147 27189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:38.128340 27189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:38.130930 27189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 262e37ee72cf4b93a419bd22126ef1e9: Sending DeleteTablet for 3 replicas of tablet 249122c944aa464ab8435154418adbb8
I20260619 02:38:38.131723 27449 tablet_service.cc:1514] Processing DeleteTablet for tablet 249122c944aa464ab8435154418adbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:38 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:55304
I20260619 02:38:38.132285 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 27252
I20260619 02:38:38.132274 27583 tablet_service.cc:1514] Processing DeleteTablet for tablet 249122c944aa464ab8435154418adbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:38 UTC) from {username='kudu', principal='kudu/127.7.194.254@KRBTEST.COM'} at 127.0.0.1:39668
I20260619 02:38:38.137801 27691 tablet_replica.cc:330] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: stopping tablet replica
I20260619 02:38:38.137827 27690 tablet_replica.cc:330] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402: stopping tablet replica
I20260619 02:38:38.138175 27691 raft_consensus.cc:2229] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:38.138191 27690 raft_consensus.cc:2229] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:38.138546 27690 raft_consensus.cc:2258] T 249122c944aa464ab8435154418adbb8 P 0c409573b63a4823816821321eafd402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:38.138547 27691 raft_consensus.cc:2258] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:38.138860 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 27386
W20260619 02:38:38.139930 27174 connection.cc:535] client connection to 127.7.194.193:41933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 02:38:38.140086 27174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:38:38.140847 27691 ts_tablet_manager.cc:1905] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:38.141614 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.141198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:41933 (local address 127.0.0.1:42346)
0619 02:38:38.141376 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:38.141460 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 02:38:38.141852 27174 catalog_manager.cc:4584] TS 6900142e2d394442b8e47ef3b1ba646d (127.7.194.193:41933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 249122c944aa464ab8435154418adbb8: Network error: Client connection negotiation failed: client connection to 127.7.194.193:41933: connect: Connection refused (error 111)
I20260619 02:38:38.143404 27691 ts_tablet_manager.cc:1918] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:38.143484 27691 log.cc:1196] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/249122c944aa464ab8435154418adbb8
I20260619 02:38:38.143801 27691 ts_tablet_manager.cc:1939] T 249122c944aa464ab8435154418adbb8 P 9a7c3f6796c143f3882dc72671fbf0a9: Deleting consensus metadata
I20260619 02:38:38.144981 27174 catalog_manager.cc:4841] TS 9a7c3f6796c143f3882dc72671fbf0a9 (127.7.194.195:32865): tablet 249122c944aa464ab8435154418adbb8 (table test-table [id=bae5c82d8ce4434c87cab3dd2bea1b59]) successfully deleted
I20260619 02:38:38.145309 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 27520
I20260619 02:38:38.146879 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.145366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:43347 (local address 127.0.0.1:55306)
0619 02:38:38.146707 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:38.146777 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.194:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1237}
I20260619 02:38:38.150822 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 27158
2026-06-19T02:38:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2416 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T02:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:38Z Disabled control of system clock
I20260619 02:38:38.181874 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:32781
--webserver_interface=127.7.194.254
--webserver_port=0
--builtin_ntp_servers=127.7.194.212:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:32781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:38:38.291968 27696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:38.292322 27696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:38.292402 27696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:38.296644 27696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:38.296753 27696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:38.296783 27696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:38.296825 27696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:38.296864 27696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:38.301970 27696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:32781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:32781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:38.303174 27696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:38.304648 27696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:38:38.313313 27696 server_base.cc:1034] running on GCE node
W20260619 02:38:38.313235 27702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:38.313239 27701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:38.313235 27704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:38.314252 27696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:38.315446 27696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:38.316617 27696 hybrid_clock.cc:648] HybridClock initialized: now 1781836718316600 us; error 29 us; skew 500 ppm
I20260619 02:38:38.318501 27696 webserver.cc:460] Webserver started at http://127.7.194.254:36465/ using document root <none> and password file <none>
I20260619 02:38:38.319099 27696 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:38.319161 27696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:38.319384 27696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:38.321487 27696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "4f05ad3db5134139b6cb9350d01443be"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "179e06fdf1a09ec164060ad0649cec49"
server_key_iv: "1a1e7ae980dd6c6a4355a0efa3c54958"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.322121 27696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "4f05ad3db5134139b6cb9350d01443be"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "179e06fdf1a09ec164060ad0649cec49"
server_key_iv: "1a1e7ae980dd6c6a4355a0efa3c54958"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.325845 27696 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:38:38.328346 27710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.329718 27696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:38.329841 27696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "4f05ad3db5134139b6cb9350d01443be"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "179e06fdf1a09ec164060ad0649cec49"
server_key_iv: "1a1e7ae980dd6c6a4355a0efa3c54958"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.329963 27696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:38.350895 27696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:38.351689 27696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:38.351902 27696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:38.358832 27696 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:32781
I20260619 02:38:38.358841 27762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:32781 every 8 connection(s)
I20260619 02:38:38.359838 27696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:38.363020 27763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:38.367817 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 27696
I20260619 02:38:38.367928 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:38.368260 7947 external_mini_cluster.cc:1428] Setting key 3db42cd7db8ab4eb4e2c20fa4eb6c663
I20260619 02:38:38.368637 27763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be: Bootstrap starting.
I20260619 02:38:38.371083 27763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:38.371884 27763 log.cc:824] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:38.374279 27763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be: No bootstrap required, opened a new log
I20260619 02:38:38.378386 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.369755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44076 (local address 127.7.194.254:32781)
0619 02:38:38.370370 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:38.370384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:38.370419 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:38:38.371037 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:38.371054 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:38.371498 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:38.371781 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:38.372176 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.373418 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.374801 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.375530 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.375730 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.376595 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:38.376602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:38.376614 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:38.376990 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:38.377048 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:38.377066 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:38.377175 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:38.377339 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 02:38:38.377642 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":140,"threads_started":1}
I20260619 02:38:38.378588 27763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05ad3db5134139b6cb9350d01443be" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 32781 } }
I20260619 02:38:38.378824 27763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:38.378907 27763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f05ad3db5134139b6cb9350d01443be, State: Initialized, Role: FOLLOWER
I20260619 02:38:38.379329 27763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05ad3db5134139b6cb9350d01443be" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 32781 } }
I20260619 02:38:38.379472 27763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:38.379590 27763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:38.379681 27763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:38.380586 27763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05ad3db5134139b6cb9350d01443be" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 32781 } }
I20260619 02:38:38.380939 27763 leader_election.cc:302] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f05ad3db5134139b6cb9350d01443be; no voters:
I20260619 02:38:38.381237 27763 leader_election.cc:288] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:38.381352 27768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:38.381613 27768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [term 1 LEADER]: Becoming Leader. State: Replica: 4f05ad3db5134139b6cb9350d01443be, State: Running, Role: LEADER
I20260619 02:38:38.381958 27768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05ad3db5134139b6cb9350d01443be" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 32781 } }
I20260619 02:38:38.382434 27763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:38.383559 27770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f05ad3db5134139b6cb9350d01443be. Latest consensus state: current_term: 1 leader_uuid: "4f05ad3db5134139b6cb9350d01443be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05ad3db5134139b6cb9350d01443be" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 32781 } } }
I20260619 02:38:38.383687 27770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:38.383948 27769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f05ad3db5134139b6cb9350d01443be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05ad3db5134139b6cb9350d01443be" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 32781 } } }
I20260619 02:38:38.384039 27777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:38.384155 27769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:38.387408 27777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:38.392841 27777 catalog_manager.cc:1347] Generated new cluster ID: 58a52ca7a4d04e6689cfbdac566d11d9
I20260619 02:38:38.392911 27777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:38.403861 27777 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:38.404851 27777 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:38.410190 27777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be: Generated new TSK 0
I20260619 02:38:38.410913 27777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:38.420362 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.193:0
--local_ip_for_outbound_sockets=127.7.194.193
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:32781
--builtin_ntp_servers=127.7.194.212:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:38:38.534081 27787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:38.534376 27787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:38.534448 27787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:38.538671 27787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:38.538733 27787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:38.538792 27787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.193
I20260619 02:38:38.543671 27787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.194.193
--webserver_port=0
--tserver_master_addrs=127.7.194.254:32781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.193
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:38.544790 27787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:38.546340 27787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:38.556216 27795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:38.556260 27792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:38.556260 27793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:38.556299 27787 server_base.cc:1034] running on GCE node
I20260619 02:38:38.556818 27787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:38.557435 27787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:38.558593 27787 hybrid_clock.cc:648] HybridClock initialized: now 1781836718558575 us; error 29 us; skew 500 ppm
I20260619 02:38:38.564801 27787 webserver.cc:460] Webserver started at http://127.7.194.193:42365/ using document root <none> and password file <none>
I20260619 02:38:38.565418 27787 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:38.565481 27787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:38.565740 27787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:38.567903 27787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/data/instance:
uuid: "93f7b8fe2da14088a076b503fa243c21"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "e82052bece5308a90fcca52779aa8fd9"
server_key_iv: "42af3c86c44b9ddecdb56bbdcbdb4e04"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.568470 27787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance:
uuid: "93f7b8fe2da14088a076b503fa243c21"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "e82052bece5308a90fcca52779aa8fd9"
server_key_iv: "42af3c86c44b9ddecdb56bbdcbdb4e04"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.572162 27787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:38:38.574718 27801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.576112 27787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:38:38.576248 27787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/wal
uuid: "93f7b8fe2da14088a076b503fa243c21"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "e82052bece5308a90fcca52779aa8fd9"
server_key_iv: "42af3c86c44b9ddecdb56bbdcbdb4e04"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.576359 27787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:38.598481 27787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:38.599256 27787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:38.599463 27787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:38.600162 27787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:38.601171 27787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:38.601243 27787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.601317 27787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:38.601361 27787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.610508 27787 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.193:33369
I20260619 02:38:38.610527 27914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.193:33369 every 8 connection(s)
I20260619 02:38:38.611485 27787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/data/info.pb
I20260619 02:38:38.620759 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 27787
I20260619 02:38:38.620905 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-0/wal/instance
I20260619 02:38:38.620999 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.613379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.193:49117 (local address 127.7.194.254:32781)
0619 02:38:38.613577 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:38.613582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:38.614657 (+ 1075us) server_negotiation.cc:407] Connection header received
0619 02:38:38.616066 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:38.616068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:38.616118 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:38.616197 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:38.617132 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.617731 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.618908 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.619136 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.619221 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.620435 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:38.620441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:38.620447 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:38.620466 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:38.620492 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:38.620505 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:38.620595 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:38.620755 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 02:38:38.620826 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:38:38.621296 7947 external_mini_cluster.cc:1428] Setting key c20a7894e479228325e68f0d5380a5f3
I20260619 02:38:38.622095 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.613842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:32781 (local address 127.7.194.193:49117)
0619 02:38:38.614498 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:38.614553 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:38.615836 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:38.616360 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:38.616368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:38.616454 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:38.616973 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.616986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.617871 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.617874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.618794 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.618804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.619274 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.619276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.619417 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.620150 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:38:38.620151 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:38.620195 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:38:38.620263 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:38.620609 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:38.620616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:38.620620 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:38:38.620716 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:38:38.621069 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260619 02:38:38.623366 27915 heartbeater.cc:347] Connected to a master server at 127.7.194.254:32781
I20260619 02:38:38.623457 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.194:0
--local_ip_for_outbound_sockets=127.7.194.194
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:32781
--builtin_ntp_servers=127.7.194.212:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:38:38.623641 27915 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:38.624276 27915 heartbeater.cc:511] Master 127.7.194.254:32781 requested a full tablet report, sending...
I20260619 02:38:38.625978 27727 ts_manager.cc:194] Registered new tserver with Master: 93f7b8fe2da14088a076b503fa243c21 (127.7.194.193:33369)
I20260619 02:38:38.627185 27727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.193:49117
W20260619 02:38:38.734012 27918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:38.734274 27918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:38.734340 27918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:38.738402 27918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:38.738457 27918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:38.738512 27918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.194
I20260619 02:38:38.743187 27918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.194.194
--webserver_port=0
--tserver_master_addrs=127.7.194.254:32781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.194
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:38.744228 27918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:38.745558 27918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:38.755095 27923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:38.755167 27924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:38.755122 27926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:38.755554 27918 server_base.cc:1034] running on GCE node
I20260619 02:38:38.755957 27918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:38.756669 27918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:38.757843 27918 hybrid_clock.cc:648] HybridClock initialized: now 1781836718757825 us; error 29 us; skew 500 ppm
I20260619 02:38:38.759732 27918 webserver.cc:460] Webserver started at http://127.7.194.194:36131/ using document root <none> and password file <none>
I20260619 02:38:38.760320 27918 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:38.760411 27918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:38.760650 27918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:38.762779 27918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/data/instance:
uuid: "6029d934b66f4e67b276308e70f88845"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "42ee31a3028acdf1caf1e30fb8269408"
server_key_iv: "f58a55f47d18a820c775c3dc766a64b4"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.763353 27918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance:
uuid: "6029d934b66f4e67b276308e70f88845"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "42ee31a3028acdf1caf1e30fb8269408"
server_key_iv: "f58a55f47d18a820c775c3dc766a64b4"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.767095 27918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:38:38.769662 27932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.770867 27918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:38.770999 27918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/wal
uuid: "6029d934b66f4e67b276308e70f88845"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "42ee31a3028acdf1caf1e30fb8269408"
server_key_iv: "f58a55f47d18a820c775c3dc766a64b4"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.771143 27918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:38.785818 27918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:38.786523 27918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:38.786716 27918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:38.787302 27918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:38.788319 27918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:38.788394 27918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.788461 27918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:38.788507 27918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.796938 27918 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.194:34851
I20260619 02:38:38.797010 28045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.194:34851 every 8 connection(s)
I20260619 02:38:38.797906 27918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/data/info.pb
I20260619 02:38:38.800187 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 27918
I20260619 02:38:38.800304 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/wal/instance
I20260619 02:38:38.800700 7947 external_mini_cluster.cc:1428] Setting key 68c41b8928a0e7dbe0dbc925920cbe22
I20260619 02:38:38.803014 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.194.195:0
--local_ip_for_outbound_sockets=127.7.194.195
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:32781
--builtin_ntp_servers=127.7.194.212:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:38:38.808485 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.799773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.194:48405 (local address 127.7.194.254:32781)
0619 02:38:38.799909 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:38.799913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:38.801245 (+ 1332us) server_negotiation.cc:407] Connection header received
0619 02:38:38.802978 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:38.802981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:38.803023 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:38.803127 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:38.804351 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.804878 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.806314 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.806537 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.806676 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.807937 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:38.807940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:38.807942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:38.807954 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:38.807973 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:38.807981 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:38.808116 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:38.808274 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 02:38:38.808342 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:38:38.809356 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.800201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:32781 (local address 127.7.194.194:48405)
0619 02:38:38.801040 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:38.801133 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:38.802747 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:38.803307 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:38.803318 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:38.803433 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:38.804174 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.804213 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.805006 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.805012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.806181 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.806191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.806713 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.806717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.806896 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.807663 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:38:38.807665 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:38.807709 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:38:38.807775 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:38.808117 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:38.808129 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:38.808137 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:38:38.808274 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 02:38:38.808574 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":142,"threads_started":1}
I20260619 02:38:38.810665 28046 heartbeater.cc:347] Connected to a master server at 127.7.194.254:32781
I20260619 02:38:38.810922 28046 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:38.811455 28046 heartbeater.cc:511] Master 127.7.194.254:32781 requested a full tablet report, sending...
I20260619 02:38:38.812548 27727 ts_manager.cc:194] Registered new tserver with Master: 6029d934b66f4e67b276308e70f88845 (127.7.194.194:34851)
I20260619 02:38:38.813446 27727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.194:48405
W20260619 02:38:38.916625 28049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:38.916883 28049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:38.916923 28049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:38.921175 28049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:38.921231 28049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:38.921317 28049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.194.195
I20260619 02:38:38.926183 28049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.194.195
--webserver_port=0
--tserver_master_addrs=127.7.194.254:32781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.194.195
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:38.927284 28049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:38.928673 28049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:38.938455 28057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:38.938462 28055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:38.938462 28054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:38.938530 28049 server_base.cc:1034] running on GCE node
I20260619 02:38:38.939078 28049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:38.939745 28049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:38.940932 28049 hybrid_clock.cc:648] HybridClock initialized: now 1781836718940911 us; error 40 us; skew 500 ppm
I20260619 02:38:38.942860 28049 webserver.cc:460] Webserver started at http://127.7.194.195:38721/ using document root <none> and password file <none>
I20260619 02:38:38.943446 28049 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:38.943547 28049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:38.943796 28049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:38.945971 28049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/data/instance:
uuid: "0349dcf682724116baa7261c794e7aa5"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "bbe134dfa4ab1c61cd9da67846685c0c"
server_key_iv: "4673b700826ae0142f9be755ef300240"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.946571 28049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance:
uuid: "0349dcf682724116baa7261c794e7aa5"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "bbe134dfa4ab1c61cd9da67846685c0c"
server_key_iv: "4673b700826ae0142f9be755ef300240"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.950448 28049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:38:38.952857 28063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.954166 28049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:38.954293 28049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/wal
uuid: "0349dcf682724116baa7261c794e7aa5"
format_stamp: "Formatted at 2026-06-19 02:38:38 on dist-test-slave-9tkz"
server_key: "bbe134dfa4ab1c61cd9da67846685c0c"
server_key_iv: "4673b700826ae0142f9be755ef300240"
server_key_version: "encryptionkey@0"
I20260619 02:38:38.954406 28049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:38.972769 28049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:38.973482 28049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:38.973704 28049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:38.974284 28049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:38:38.975338 28049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:38:38.975405 28049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.975472 28049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:38:38.975521 28049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:38.985707 28049 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.195:35951
I20260619 02:38:38.985750 28176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.195:35951 every 8 connection(s)
I20260619 02:38:38.986681 28049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/data/info.pb
I20260619 02:38:38.990753 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 28049
I20260619 02:38:38.990896 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/wal/instance
I20260619 02:38:38.991281 7947 external_mini_cluster.cc:1428] Setting key 91cb1ef58e81364be7b78c526c427626
I20260619 02:38:38.996630 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.988622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:54645 (local address 127.7.194.254:32781)
0619 02:38:38.988775 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:38.988779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:38.989873 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 02:38:38.991676 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:38.991680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:38.991744 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:38.991854 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:38.993046 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.993772 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.994905 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.995114 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.995189 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.996231 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:38.996234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:38.996236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:38.996247 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:38.996264 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:38.996271 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:38.996327 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:38.996460 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 02:38:38.996508 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:38:38.997465 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:38.989048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:32781 (local address 127.7.194.195:54645)
0619 02:38:38.989719 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:38.989759 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:38.991429 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:38.992115 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:38.992122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:38.992254 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:38:38.992907 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.992923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.993891 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.993894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.994801 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:38.994810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:38.995243 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:38.995246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:38.995393 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:38.995977 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:38:38.995979 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:38:38.996022 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:38:38.996087 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:38:38.996342 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:38:38.996348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:38:38.996352 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:38:38.996446 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:38:38.996675 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260619 02:38:38.998579 28177 heartbeater.cc:347] Connected to a master server at 127.7.194.254:32781
I20260619 02:38:38.998819 28177 heartbeater.cc:464] Registering TS with master...
I20260619 02:38:38.999365 28177 heartbeater.cc:511] Master 127.7.194.254:32781 requested a full tablet report, sending...
I20260619 02:38:39.000435 27727 ts_manager.cc:194] Registered new tserver with Master: 0349dcf682724116baa7261c794e7aa5 (127.7.194.195:35951)
I20260619 02:38:39.001055 27727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.194.195:54645
I20260619 02:38:39.005304 7947 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:38:39.010777 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.007246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.7.194.254:32781)
0619 02:38:39.007415 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:39.007419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:39.007522 (+ 103us) server_negotiation.cc:407] Connection header received
0619 02:38:39.007631 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:39.007633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:39.007670 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:39.007733 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:39.007981 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.008502 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.009338 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.009544 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.009673 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.010370 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:39.010372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:39.010374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:39.010384 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:39.010401 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:39.010409 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:39.010464 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:39.010567 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 02:38:39.010611 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 02:38:39.013517 27727 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:38:39.016439 27727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:38:39.030885 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.025517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:33369 (local address 127.0.0.1:58022)
0619 02:38:39.025818 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:39.025829 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:39.025914 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:39.026966 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:39.026969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:39.026987 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:39.027096 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.027106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.028052 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.028058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.029291 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.029300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.030355 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.030358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.030469 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.030523 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:38:39.030625 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:38:39.030741 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":6,"thread_start_us":123,"threads_started":1}
I20260619 02:38:39.031040 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.024818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:34851 (local address 127.0.0.1:59036)
0619 02:38:39.025339 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:39.025388 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:39.025510 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:39.026687 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:39.026693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:39.026717 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:39.026846 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.026852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.028341 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.028344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.029666 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.029676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.030626 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.030629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.030752 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.030775 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:38:39.030838 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:38:39.030905 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":93,"thread_start_us":75,"threads_started":1}
I20260619 02:38:39.031117 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.024974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.7.194.194:34851)
0619 02:38:39.026145 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:39.026159 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:39.026186 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:38:39.026267 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:39.026273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:39.026490 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:39.026670 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:39.026990 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.028225 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.029797 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.030524 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.030624 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.030806 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:39.030864 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 02:38:39.030953 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020,"thread_start_us":127,"threads_started":1}
I20260619 02:38:39.031524 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.026175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.7.194.193:33369)
0619 02:38:39.026522 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:39.026533 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:39.026556 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:38:39.026643 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:39.026648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:39.026820 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:39.026989 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:39.027196 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.027938 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.029414 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.030229 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.030338 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.030533 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:39.031292 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 02:38:39.031387 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":127,"threads_started":1}
I20260619 02:38:39.031632 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.025748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.7.194.195:35951)
0619 02:38:39.026581 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:39.026590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:39.026634 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:38:39.026707 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:39.026712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:39.026904 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:39.027296 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:39.027432 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.028519 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.029920 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.030510 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.030740 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.030907 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:39.031374 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 02:38:39.031475 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":168,"threads_started":1}
I20260619 02:38:39.031513 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.024977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.195:35951 (local address 127.0.0.1:36822)
0619 02:38:39.025747 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:39.025759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:39.025836 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:39.027053 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:39.027056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:39.027077 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:39.027154 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.027160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.028635 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.028638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.029793 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.029807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.030643 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.030646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.031212 (+ 566us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.031249 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 02:38:39.031328 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:38:39.031386 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":447,"thread_start_us":137,"threads_started":1}
I20260619 02:38:39.034324 27849 tablet_service.cc:1467] Processing CreateTablet for tablet 252fcd1b0ed54768880ba6c5972236c5 (DEFAULT_TABLE table=test-table [id=eec43f0433b34b4695d2578d7d7ff9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:39.034324 27980 tablet_service.cc:1467] Processing CreateTablet for tablet 252fcd1b0ed54768880ba6c5972236c5 (DEFAULT_TABLE table=test-table [id=eec43f0433b34b4695d2578d7d7ff9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:39.034608 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 252fcd1b0ed54768880ba6c5972236c5 (DEFAULT_TABLE table=test-table [id=eec43f0433b34b4695d2578d7d7ff9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:38:39.035445 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252fcd1b0ed54768880ba6c5972236c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:39.035446 27980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252fcd1b0ed54768880ba6c5972236c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:39.035446 27849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252fcd1b0ed54768880ba6c5972236c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:39.041491 28194 tablet_bootstrap.cc:492] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: Bootstrap starting.
I20260619 02:38:39.043154 28196 tablet_bootstrap.cc:492] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: Bootstrap starting.
I20260619 02:38:39.043161 28195 tablet_bootstrap.cc:492] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21: Bootstrap starting.
I20260619 02:38:39.044698 28194 tablet_bootstrap.cc:654] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:39.044844 28196 tablet_bootstrap.cc:654] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:39.045145 28195 tablet_bootstrap.cc:654] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:39.045588 28194 log.cc:824] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:39.045606 28196 log.cc:824] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:39.045929 28195 log.cc:824] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:39.047574 28194 tablet_bootstrap.cc:492] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: No bootstrap required, opened a new log
I20260619 02:38:39.047574 28196 tablet_bootstrap.cc:492] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: No bootstrap required, opened a new log
I20260619 02:38:39.047765 28194 ts_tablet_manager.cc:1397] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:38:39.047765 28196 ts_tablet_manager.cc:1397] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:38:39.048125 28195 tablet_bootstrap.cc:492] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21: No bootstrap required, opened a new log
I20260619 02:38:39.048305 28195 ts_tablet_manager.cc:1397] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:38:39.051271 28195 raft_consensus.cc:348] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } }
I20260619 02:38:39.051508 28195 raft_consensus.cc:374] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:39.051580 28195 raft_consensus.cc:729] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7b8fe2da14088a076b503fa243c21, State: Initialized, Role: FOLLOWER
I20260619 02:38:39.052033 28195 consensus_queue.cc:260] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } }
I20260619 02:38:39.052121 28194 raft_consensus.cc:348] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } }
I20260619 02:38:39.052117 28196 raft_consensus.cc:348] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } }
I20260619 02:38:39.052361 28194 raft_consensus.cc:374] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:39.052361 28196 raft_consensus.cc:374] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:39.052416 28196 raft_consensus.cc:729] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6029d934b66f4e67b276308e70f88845, State: Initialized, Role: FOLLOWER
I20260619 02:38:39.052424 28194 raft_consensus.cc:729] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0349dcf682724116baa7261c794e7aa5, State: Initialized, Role: FOLLOWER
I20260619 02:38:39.052789 27915 heartbeater.cc:503] Master 127.7.194.254:32781 was elected leader, sending a full tablet report...
I20260619 02:38:39.052884 28196 consensus_queue.cc:260] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } }
I20260619 02:38:39.052884 28194 consensus_queue.cc:260] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } }
I20260619 02:38:39.053027 28195 ts_tablet_manager.cc:1428] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:38:39.053841 28194 ts_tablet_manager.cc:1428] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:38:39.053841 28196 ts_tablet_manager.cc:1428] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:38:39.054162 28046 heartbeater.cc:503] Master 127.7.194.254:32781 was elected leader, sending a full tablet report...
I20260619 02:38:39.055709 28177 heartbeater.cc:503] Master 127.7.194.254:32781 was elected leader, sending a full tablet report...
W20260619 02:38:39.112805 27916 tablet.cc:2369] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:38:39.129312 28202 raft_consensus.cc:482] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:38:39.129500 28202 raft_consensus.cc:504] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } }
I20260619 02:38:39.130688 28202 leader_election.cc:288] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6029d934b66f4e67b276308e70f88845 (127.7.194.194:34851), 93f7b8fe2da14088a076b503fa243c21 (127.7.194.193:33369)
I20260619 02:38:39.135141 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.130910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.194:34851 (local address 127.7.194.195:55485)
0619 02:38:39.131033 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:39.131050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:39.131158 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:39.131420 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:39.131423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:39.131442 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:39.131564 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.131570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.132615 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.132621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.133910 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.133920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.134713 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.134716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.134820 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.134850 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:38:39.134924 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:38:39.134998 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260619 02:38:39.135221 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.131006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:55485 (local address 127.7.194.194:34851)
0619 02:38:39.131157 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:39.131163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:39.131181 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:38:39.131245 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:39.131247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:39.131301 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:39.131395 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:39.131673 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.132487 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.134021 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.134598 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.134713 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.134762 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:39.134973 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 02:38:39.135072 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:38:39.135499 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.131347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.194.195:53309 (local address 127.7.194.193:33369)
0619 02:38:39.131471 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:39.131476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:39.131740 (+ 264us) server_negotiation.cc:407] Connection header received
0619 02:38:39.131813 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:39.131816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:39.131867 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:39.131947 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:38:39.132192 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.132822 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.134308 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.134921 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.135032 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.135073 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:39.135314 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 02:38:39.135412 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:38:39.135887 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.131200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:33369 (local address 127.7.194.195:53309)
0619 02:38:39.131649 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:39.131664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:38:39.131743 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:38:39.131960 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:38:39.131962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:38:39.131973 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:38:39.132084 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.132089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.132926 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.132929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.134187 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:38:39.134198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.135048 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.135051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:38:39.135146 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.135177 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:38:39.135670 (+ 493us) client_negotiation.cc:240] Negotiation successful
0619 02:38:39.135746 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":21,"thread_start_us":146,"threads_started":1}
I20260619 02:38:39.136040 28000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252fcd1b0ed54768880ba6c5972236c5" candidate_uuid: "0349dcf682724116baa7261c794e7aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6029d934b66f4e67b276308e70f88845" is_pre_election: true
I20260619 02:38:39.136384 28000 raft_consensus.cc:2454] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0349dcf682724116baa7261c794e7aa5 in term 0.
I20260619 02:38:39.136503 27869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252fcd1b0ed54768880ba6c5972236c5" candidate_uuid: "0349dcf682724116baa7261c794e7aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7b8fe2da14088a076b503fa243c21" is_pre_election: true
I20260619 02:38:39.136842 27869 raft_consensus.cc:2454] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0349dcf682724116baa7261c794e7aa5 in term 0.
I20260619 02:38:39.136890 28067 leader_election.cc:302] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0349dcf682724116baa7261c794e7aa5, 6029d934b66f4e67b276308e70f88845; no voters:
I20260619 02:38:39.137192 28202 raft_consensus.cc:2781] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:38:39.137266 28202 raft_consensus.cc:482] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:38:39.137316 28202 raft_consensus.cc:3037] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:39.138391 28202 raft_consensus.cc:504] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } }
I20260619 02:38:39.138731 28202 leader_election.cc:288] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [CANDIDATE]: Term 1 election: Requested vote from peers 6029d934b66f4e67b276308e70f88845 (127.7.194.194:34851), 93f7b8fe2da14088a076b503fa243c21 (127.7.194.193:33369)
I20260619 02:38:39.139096 28000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252fcd1b0ed54768880ba6c5972236c5" candidate_uuid: "0349dcf682724116baa7261c794e7aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6029d934b66f4e67b276308e70f88845"
I20260619 02:38:39.139149 27869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252fcd1b0ed54768880ba6c5972236c5" candidate_uuid: "0349dcf682724116baa7261c794e7aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7b8fe2da14088a076b503fa243c21"
I20260619 02:38:39.139240 28000 raft_consensus.cc:3037] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:39.139266 27869 raft_consensus.cc:3037] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:39.140251 27869 raft_consensus.cc:2454] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0349dcf682724116baa7261c794e7aa5 in term 1.
I20260619 02:38:39.140262 28000 raft_consensus.cc:2454] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0349dcf682724116baa7261c794e7aa5 in term 1.
I20260619 02:38:39.140527 28067 leader_election.cc:302] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0349dcf682724116baa7261c794e7aa5, 93f7b8fe2da14088a076b503fa243c21; no voters:
I20260619 02:38:39.140798 28202 raft_consensus.cc:2781] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:39.141064 28202 raft_consensus.cc:686] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 0349dcf682724116baa7261c794e7aa5, State: Running, Role: LEADER
I20260619 02:38:39.141395 28202 consensus_queue.cc:237] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } }
I20260619 02:38:39.144816 27726 catalog_manager.cc:5495] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0349dcf682724116baa7261c794e7aa5 (127.7.194.195). New cstate: current_term: 1 leader_uuid: "0349dcf682724116baa7261c794e7aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0349dcf682724116baa7261c794e7aa5" member_type: VOTER last_known_addr { host: "127.7.194.195" port: 35951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20260619 02:38:39.184332 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.180787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.7.194.195:35951)
0619 02:38:39.180918 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:39.180921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:39.180935 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:38:39.181014 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:39.181016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:39.181079 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:39.181184 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:38:39.181400 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.181958 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.182918 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:39.183164 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:39.183297 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:39.183573 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:38:39.184004 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:38:39.184125 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:39.184155 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 02:38:39.184200 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:38:39.190672 28000 raft_consensus.cc:1261] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [term 1 FOLLOWER]: Refusing update from remote peer 0349dcf682724116baa7261c794e7aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:39.190672 27869 raft_consensus.cc:1261] T 252fcd1b0ed54768880ba6c5972236c5 P 93f7b8fe2da14088a076b503fa243c21 [term 1 FOLLOWER]: Refusing update from remote peer 0349dcf682724116baa7261c794e7aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:38:39.191437 28204 consensus_queue.cc:1035] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6029d934b66f4e67b276308e70f88845" member_type: VOTER last_known_addr { host: "127.7.194.194" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:39.191684 28202 consensus_queue.cc:1035] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7b8fe2da14088a076b503fa243c21" member_type: VOTER last_known_addr { host: "127.7.194.193" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:38:39.199110 28207 mvcc.cc:205] Tried to move back new op lower bound from 7298403201796714496 to 7298403201606311936. Current Snapshot: MvccSnapshot[applied={T|T < 7298403201796714496}]
I20260619 02:38:39.200591 28208 mvcc.cc:205] Tried to move back new op lower bound from 7298403201796714496 to 7298403201606311936. Current Snapshot: MvccSnapshot[applied={T|T < 7298403201796714496}]
I20260619 02:38:39.201282 28209 mvcc.cc:205] Tried to move back new op lower bound from 7298403201796714496 to 7298403201606311936. Current Snapshot: MvccSnapshot[applied={T|T < 7298403201796714496}]
I20260619 02:38:39.208038 27726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:39.208240 27726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:38:39.210752 27726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f05ad3db5134139b6cb9350d01443be: Sending DeleteTablet for 3 replicas of tablet 252fcd1b0ed54768880ba6c5972236c5
I20260619 02:38:39.211643 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 252fcd1b0ed54768880ba6c5972236c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:39 UTC) from {username='slave'} at 127.0.0.1:36822
I20260619 02:38:39.211637 27849 tablet_service.cc:1514] Processing DeleteTablet for tablet 252fcd1b0ed54768880ba6c5972236c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:39 UTC) from {username='slave'} at 127.0.0.1:58022
I20260619 02:38:39.211635 27980 tablet_service.cc:1514] Processing DeleteTablet for tablet 252fcd1b0ed54768880ba6c5972236c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:38:39 UTC) from {username='slave'} at 127.0.0.1:59036
I20260619 02:38:39.212215 28218 tablet_replica.cc:330] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: stopping tablet replica
I20260619 02:38:39.212602 28218 raft_consensus.cc:2229] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:38:39.212898 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 27787
I20260619 02:38:39.212965 28218 raft_consensus.cc:2258] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:39.213363 28219 tablet_replica.cc:330] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: stopping tablet replica
I20260619 02:38:39.213680 28219 raft_consensus.cc:2229] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:38:39.214027 28219 raft_consensus.cc:2258] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:38:39.215448 28219 ts_tablet_manager.cc:1905] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:39.216841 28218 ts_tablet_manager.cc:1905] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:38:39.218578 28219 ts_tablet_manager.cc:1918] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:39.218698 28219 log.cc:1196] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-2/wal/wals/252fcd1b0ed54768880ba6c5972236c5
I20260619 02:38:39.219066 28219 ts_tablet_manager.cc:1939] T 252fcd1b0ed54768880ba6c5972236c5 P 0349dcf682724116baa7261c794e7aa5: Deleting consensus metadata
I20260619 02:38:39.220023 28218 ts_tablet_manager.cc:1918] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:38:39.220192 28218 log.cc:1196] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: Deleting WAL directory at /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781836464269074-7947-0/minicluster-data/ts-1/wal/wals/252fcd1b0ed54768880ba6c5972236c5
I20260619 02:38:39.220430 27712 catalog_manager.cc:4841] TS 0349dcf682724116baa7261c794e7aa5 (127.7.194.195:35951): tablet 252fcd1b0ed54768880ba6c5972236c5 (table test-table [id=eec43f0433b34b4695d2578d7d7ff9a0]) successfully deleted
I20260619 02:38:39.220696 28218 ts_tablet_manager.cc:1939] T 252fcd1b0ed54768880ba6c5972236c5 P 6029d934b66f4e67b276308e70f88845: Deleting consensus metadata
I20260619 02:38:39.221835 27714 catalog_manager.cc:4841] TS 6029d934b66f4e67b276308e70f88845 (127.7.194.194:34851): tablet 252fcd1b0ed54768880ba6c5972236c5 (table test-table [id=eec43f0433b34b4695d2578d7d7ff9a0]) successfully deleted
W20260619 02:38:39.222347 27714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.194.193:33369 (error 108)
I20260619 02:38:39.222558 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 27918
I20260619 02:38:39.223694 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:39.223365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.193:33369 (local address 127.0.0.1:58028)
0619 02:38:39.223531 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 02:38:39.223608 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.193:33369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 02:38:39.223888 27714 catalog_manager.cc:4584] TS 93f7b8fe2da14088a076b503fa243c21 (127.7.194.193:33369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 252fcd1b0ed54768880ba6c5972236c5: Network error: Client connection negotiation failed: client connection to 127.7.194.193:33369: connect: Connection refused (error 111)
I20260619 02:38:39.227921 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 28049
I20260619 02:38:39.233187 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 27696
2026-06-19T02:38:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1082 ms)
[----------] 6 tests from EncryptionPolicyTest (10932 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:39 dist-test-slave-9tkz krb5kdc[28223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:39 dist-test-slave-9tkz krb5kdc[28223](info): set up 2 sockets
Jun 19 02:38:39 dist-test-slave-9tkz krb5kdc[28223](info): commencing operation
krb5kdc: starting...
W20260619 02:38:40.269675 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:40 dist-test-slave-9tkz krb5kdc[28223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:40.388090 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:40.500718 28236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:40.501003 28236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:40.501047 28236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:40.505208 28236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:40.505271 28236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:40.505290 28236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:40.505307 28236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:40.505322 28236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:40.510629 28236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:40.511665 28236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:40.512701 28236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:40.521384 28244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:40.521384 28242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:40.521401 28241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:40.521510 28236 server_base.cc:1034] running on GCE node
I20260619 02:38:40.522320 28236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:40.523382 28236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:40.524559 28236 hybrid_clock.cc:648] HybridClock initialized: now 1781836720524549 us; error 41 us; skew 500 ppm
Jun 19 02:38:40 dist-test-slave-9tkz krb5kdc[28223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:40.527457 28236 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:40.528402 28236 webserver.cc:460] Webserver started at http://127.0.0.1:36561/ using document root <none> and password file <none>
I20260619 02:38:40.528995 28236 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:40.529085 28236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:40.529332 28236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:40.531471 28236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "ec794e54592248338a9e0cb019c9cb0f"
format_stamp: "Formatted at 2026-06-19 02:38:40 on dist-test-slave-9tkz"
server_key: "921a5cedf4cbcc0c13263c978b60505e"
server_key_iv: "94dc60a9bcd601016f86503e7f51020b"
server_key_version: "encryptionkey@0"
I20260619 02:38:40.532084 28236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "ec794e54592248338a9e0cb019c9cb0f"
format_stamp: "Formatted at 2026-06-19 02:38:40 on dist-test-slave-9tkz"
server_key: "921a5cedf4cbcc0c13263c978b60505e"
server_key_iv: "94dc60a9bcd601016f86503e7f51020b"
server_key_version: "encryptionkey@0"
I20260619 02:38:40.535929 28236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:38:40.538335 28251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:40.539551 28236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:40.539695 28236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "ec794e54592248338a9e0cb019c9cb0f"
format_stamp: "Formatted at 2026-06-19 02:38:40 on dist-test-slave-9tkz"
server_key: "921a5cedf4cbcc0c13263c978b60505e"
server_key_iv: "94dc60a9bcd601016f86503e7f51020b"
server_key_version: "encryptionkey@0"
I20260619 02:38:40.539806 28236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:40.555305 28236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:40.558413 28236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:40.558629 28236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:40.565634 28236 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36775
I20260619 02:38:40.565658 28303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36775 every 8 connection(s)
I20260619 02:38:40.566608 28236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:40.569329 28304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:40.574544 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 28236
I20260619 02:38:40.574724 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:40.575047 7947 external_mini_cluster.cc:1428] Setting key b83076c7dee1e626390c16bda14a7a74
I20260619 02:38:40.575142 28304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f: Bootstrap starting.
I20260619 02:38:40.577749 28304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:40.578524 28304 log.cc:824] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:40.580816 28304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f: No bootstrap required, opened a new log
I20260619 02:38:40.585309 28304 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec794e54592248338a9e0cb019c9cb0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260619 02:38:40.585552 28304 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:40.585625 28304 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec794e54592248338a9e0cb019c9cb0f, State: Initialized, Role: FOLLOWER
I20260619 02:38:40.586143 28304 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec794e54592248338a9e0cb019c9cb0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260619 02:38:40.586297 28304 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:40.586355 28304 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 19 02:38:40 dist-test-slave-9tkz krb5kdc[28223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:38:40.586458 28304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:40.587714 28304 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec794e54592248338a9e0cb019c9cb0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260619 02:38:40.588120 28304 leader_election.cc:302] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec794e54592248338a9e0cb019c9cb0f; no voters:
I20260619 02:38:40.588491 28304 leader_election.cc:288] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:40.588881 28310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:40.589099 28310 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [term 1 LEADER]: Becoming Leader. State: Replica: ec794e54592248338a9e0cb019c9cb0f, State: Running, Role: LEADER
I20260619 02:38:40.589450 28310 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec794e54592248338a9e0cb019c9cb0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260619 02:38:40.589912 28304 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:40.590848 28310 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec794e54592248338a9e0cb019c9cb0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec794e54592248338a9e0cb019c9cb0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260619 02:38:40.591049 28310 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:40.591378 28310 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec794e54592248338a9e0cb019c9cb0f. Latest consensus state: current_term: 1 leader_uuid: "ec794e54592248338a9e0cb019c9cb0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec794e54592248338a9e0cb019c9cb0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260619 02:38:40.591555 28310 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:40.591894 28318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:40.594105 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:40.576429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51077 (local address 127.0.0.1:36775)
0619 02:38:40.576857 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:40.576873 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:40.576907 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:38:40.577618 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:40.577637 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:40.578061 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:40.578397 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:40.579344 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:40.580825 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:40.582163 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:40.582895 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:40.583076 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:40.587146 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:40.587153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:40.587170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:40.587203 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:40.589410 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:40.590122 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:40.590128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:40.590134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:40.590285 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:40.590762 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:40.590766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:40.590767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:40.591361 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:40.591562 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:40.592565 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0619 02:38:40.592766 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
I20260619 02:38:40.595428 28318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:40.597451 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:40.601707 28318 catalog_manager.cc:1347] Generated new cluster ID: 62b10946d0d64d8882e955960ac0ea29
I20260619 02:38:40.601769 28318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:40.609138 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:40.599696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38293 (local address 127.0.0.1:36775)
0619 02:38:40.599942 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:40.599947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:40.600016 (+ 69us) server_negotiation.cc:407] Connection header received
0619 02:38:40.600187 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:40.600191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:40.600239 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:40.600323 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:40.601171 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:40.602074 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:40.603269 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:40.603655 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:40.603763 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:40.605401 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:40.605405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:40.605408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:40.605433 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:40.607126 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:40.607656 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:40.607659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:40.607661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:40.607743 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:40.608058 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:40.608062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:40.608063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:40.608301 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:40.608437 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:40.608714 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 02:38:40.608978 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260619 02:38:40.619673 28318 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:40.620965 28318 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:40.629904 28318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec794e54592248338a9e0cb019c9cb0f: Generated new TSK 0
I20260619 02:38:40.630795 28318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:40.636085 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 28236
W20260619 02:38:40.641495 28221 connection.cc:535] client connection to 127.0.0.1:36775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36775): Connection reset by peer (error 104)
2026-06-19T02:38:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:40 dist-test-slave-9tkz krb5kdc[28335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:40 dist-test-slave-9tkz krb5kdc[28335](info): set up 2 sockets
Jun 19 02:38:40 dist-test-slave-9tkz krb5kdc[28335](info): commencing operation
krb5kdc: starting...
W20260619 02:38:41.665781 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:41 dist-test-slave-9tkz krb5kdc[28335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:41.784332 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:41.893611 28348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:41.893958 28348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:41.894053 28348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:41.898219 28348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:41.898300 28348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:41.898348 28348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:41.898402 28348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:41.898439 28348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:41.903723 28348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:41.904814 28348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:41.905947 28348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:41.914404 28356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:41.914404 28353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:41.914424 28354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:41.914716 28348 server_base.cc:1034] running on GCE node
I20260619 02:38:41.915287 28348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:41.916388 28348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:41.917625 28348 hybrid_clock.cc:648] HybridClock initialized: now 1781836721917591 us; error 48 us; skew 500 ppm
Jun 19 02:38:41 dist-test-slave-9tkz krb5kdc[28335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:41.920437 28348 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:41.921396 28348 webserver.cc:460] Webserver started at http://127.0.0.1:41983/ using document root <none> and password file <none>
I20260619 02:38:41.922071 28348 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:41.922161 28348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:41.922379 28348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:41.924415 28348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "390a95d5994f43c5baaa65448d69d22b"
format_stamp: "Formatted at 2026-06-19 02:38:41 on dist-test-slave-9tkz"
server_key: "7c28054e562b9a826eb26f3eee298aa9"
server_key_iv: "1e4e7a36381950aa479d806a76e61dc7"
server_key_version: "encryptionkey@0"
I20260619 02:38:41.924995 28348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "390a95d5994f43c5baaa65448d69d22b"
format_stamp: "Formatted at 2026-06-19 02:38:41 on dist-test-slave-9tkz"
server_key: "7c28054e562b9a826eb26f3eee298aa9"
server_key_iv: "1e4e7a36381950aa479d806a76e61dc7"
server_key_version: "encryptionkey@0"
I20260619 02:38:41.928730 28348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:38:41.931214 28363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:41.932381 28348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 02:38:41.932528 28348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "390a95d5994f43c5baaa65448d69d22b"
format_stamp: "Formatted at 2026-06-19 02:38:41 on dist-test-slave-9tkz"
server_key: "7c28054e562b9a826eb26f3eee298aa9"
server_key_iv: "1e4e7a36381950aa479d806a76e61dc7"
server_key_version: "encryptionkey@0"
I20260619 02:38:41.932646 28348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:41.959885 28348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:41.963389 28348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:41.963554 28348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:41.970563 28348 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40343
I20260619 02:38:41.970577 28415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40343 every 8 connection(s)
I20260619 02:38:41.971491 28348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:41.974284 28416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:41.979656 28416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b: Bootstrap starting.
I20260619 02:38:41.980504 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 28348
I20260619 02:38:41.980662 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:41.980978 7947 external_mini_cluster.cc:1428] Setting key 56022f647c01b0a844984514c403a083
I20260619 02:38:41.982530 28416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:41.983394 28416 log.cc:824] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:41.985527 28416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b: No bootstrap required, opened a new log
I20260619 02:38:41.989825 28416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390a95d5994f43c5baaa65448d69d22b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40343 } }
I20260619 02:38:41.990108 28416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:41.990185 28416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390a95d5994f43c5baaa65448d69d22b, State: Initialized, Role: FOLLOWER
I20260619 02:38:41.990674 28416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390a95d5994f43c5baaa65448d69d22b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40343 } }
I20260619 02:38:41.990828 28416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:41.990886 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:41.982181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41455 (local address 127.0.0.1:40343)
0619 02:38:41.982650 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:41.982672 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:41.982719 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:38:41.983580 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:41.983605 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:41.983992 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:41.984348 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:41.984760 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:41.986179 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:41.987057 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:41.987750 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:41.987911 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:41.988766 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:41.988772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:41.988789 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:41.989296 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:41.989364 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:41.989395 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:41.989526 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:41.989749 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 02:38:41.989887 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260619 02:38:41.990893 28416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:41.991014 28416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:41.992141 28416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390a95d5994f43c5baaa65448d69d22b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40343 } }
I20260619 02:38:41.992507 28416 leader_election.cc:302] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 390a95d5994f43c5baaa65448d69d22b; no voters:
I20260619 02:38:41.992861 28416 leader_election.cc:288] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:41.992965 28421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:41.993209 28421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [term 1 LEADER]: Becoming Leader. State: Replica: 390a95d5994f43c5baaa65448d69d22b, State: Running, Role: LEADER
I20260619 02:38:41.993620 28421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390a95d5994f43c5baaa65448d69d22b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40343 } }
I20260619 02:38:41.994215 28416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:41.994536 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:41.997056 28423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 390a95d5994f43c5baaa65448d69d22b. Latest consensus state: current_term: 1 leader_uuid: "390a95d5994f43c5baaa65448d69d22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390a95d5994f43c5baaa65448d69d22b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40343 } } }
I20260619 02:38:41.997416 28423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:41.997233 28422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "390a95d5994f43c5baaa65448d69d22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390a95d5994f43c5baaa65448d69d22b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40343 } } }
I20260619 02:38:41.997682 28422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b [sys.catalog]: This master's current role is: LEADER
W20260619 02:38:42.000082 28440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:38:42.000169 28440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:38:42.000267 28442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:42.000952 28442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:42.002624 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:41.996892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54219 (local address 127.0.0.1:40343)
0619 02:38:41.996989 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:41.996994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:41.998013 (+ 1019us) server_negotiation.cc:407] Connection header received
0619 02:38:41.998111 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:41.998114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:41.998159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:41.998237 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:41.998475 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:41.999248 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:42.000401 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:42.000757 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:42.000849 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:42.002080 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:42.002084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:42.002086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:42.002101 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:42.002129 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:42.002143 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:42.002204 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:42.002448 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 02:38:42.002485 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:38:42.006798 28442 catalog_manager.cc:1347] Generated new cluster ID: bc7599f1698f4a3d88c2bb0cbb617f53
I20260619 02:38:42.006875 28442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:42.024078 28442 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:42.025434 28442 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:42.031375 28442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 390a95d5994f43c5baaa65448d69d22b: Generated new TSK 0
I20260619 02:38:42.032047 28442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:42.037442 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 28348
W20260619 02:38:42.042899 28333 connection.cc:535] client connection to 127.0.0.1:40343 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40343): Connection reset by peer (error 104)
2026-06-19T02:38:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:42 dist-test-slave-9tkz krb5kdc[28447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:42 dist-test-slave-9tkz krb5kdc[28447](info): set up 2 sockets
Jun 19 02:38:42 dist-test-slave-9tkz krb5kdc[28447](info): commencing operation
krb5kdc: starting...
W20260619 02:38:43.085842 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:43 dist-test-slave-9tkz krb5kdc[28447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:43.203899 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:43.314527 28460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:43.314790 28460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:43.314859 28460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:43.318859 28460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:43.318943 28460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:43.318979 28460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:43.319011 28460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:43.319041 28460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:43.324262 28460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39029
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:43.325225 28460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:43.326197 28460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:43.334352 28466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:43.334487 28465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:43.334391 28468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:43.334388 28460 server_base.cc:1034] running on GCE node
I20260619 02:38:43.335117 28460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:43.336118 28460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:43.337306 28460 hybrid_clock.cc:648] HybridClock initialized: now 1781836723337294 us; error 33 us; skew 500 ppm
Jun 19 02:38:43 dist-test-slave-9tkz krb5kdc[28447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:43.340138 28460 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:43.341114 28460 webserver.cc:460] Webserver started at http://127.0.0.1:34923/ using document root <none> and password file <none>
I20260619 02:38:43.341735 28460 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:43.341827 28460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:43.342052 28460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:43.344075 28460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "80d9341e036b4684b00bc30c6150ef9c"
format_stamp: "Formatted at 2026-06-19 02:38:43 on dist-test-slave-9tkz"
server_key: "fae1abba08a3b891522660ec56393fe2"
server_key_iv: "ab8e96fb3061d9f98f80a61eeab912b6"
server_key_version: "encryptionkey@0"
I20260619 02:38:43.344638 28460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "80d9341e036b4684b00bc30c6150ef9c"
format_stamp: "Formatted at 2026-06-19 02:38:43 on dist-test-slave-9tkz"
server_key: "fae1abba08a3b891522660ec56393fe2"
server_key_iv: "ab8e96fb3061d9f98f80a61eeab912b6"
server_key_version: "encryptionkey@0"
I20260619 02:38:43.348202 28460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 02:38:43.350538 28475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:43.351855 28460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:43.352044 28460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "80d9341e036b4684b00bc30c6150ef9c"
format_stamp: "Formatted at 2026-06-19 02:38:43 on dist-test-slave-9tkz"
server_key: "fae1abba08a3b891522660ec56393fe2"
server_key_iv: "ab8e96fb3061d9f98f80a61eeab912b6"
server_key_version: "encryptionkey@0"
I20260619 02:38:43.352166 28460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:43.356848 28460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:43.360126 28460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:43.360324 28460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:43.367159 28460 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39029
I20260619 02:38:43.367172 28527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39029 every 8 connection(s)
I20260619 02:38:43.368044 28460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:43.369844 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 28460
I20260619 02:38:43.369995 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:43.370311 7947 external_mini_cluster.cc:1428] Setting key d0cb8190228992bb780c4ac67c1315c8
I20260619 02:38:43.371145 28528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:43.375335 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:43.371617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42487 (local address 127.0.0.1:39029)
0619 02:38:43.372059 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:43.372074 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:43.372108 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:38:43.372693 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:43.372709 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:43.372979 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:43.373238 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:43.374142 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:43.374147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:43.374158 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:43.374168 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:43.374223 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:43.374242 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:43.374354 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:43.374468 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 02:38:43.374646 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260619 02:38:43.377482 28528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c: Bootstrap starting.
I20260619 02:38:43.378108 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:43.379722 28528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:43.380625 28528 log.cc:824] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:43.381755 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:43.380106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33815 (local address 127.0.0.1:39029)
0619 02:38:43.380209 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:43.380214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:43.380358 (+ 144us) server_negotiation.cc:407] Connection header received
0619 02:38:43.380515 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:43.380517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:43.380548 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:43.380616 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:43.381326 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:43.381328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:43.381331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:43.381341 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:43.381364 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:43.381374 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:43.381463 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:43.381594 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 02:38:43.381637 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:38:43.382872 28528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c: No bootstrap required, opened a new log
I20260619 02:38:43.386441 28528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d9341e036b4684b00bc30c6150ef9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260619 02:38:43.386646 28528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:43.386686 28528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d9341e036b4684b00bc30c6150ef9c, State: Initialized, Role: FOLLOWER
I20260619 02:38:43.387105 28528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d9341e036b4684b00bc30c6150ef9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260619 02:38:43.387224 28528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:43.387270 28528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:43.387341 28528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:43.388305 28528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d9341e036b4684b00bc30c6150ef9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260619 02:38:43.388586 28528 leader_election.cc:302] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80d9341e036b4684b00bc30c6150ef9c; no voters:
I20260619 02:38:43.388837 28528 leader_election.cc:288] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:43.388926 28538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:43.389156 28538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [term 1 LEADER]: Becoming Leader. State: Replica: 80d9341e036b4684b00bc30c6150ef9c, State: Running, Role: LEADER
I20260619 02:38:43.389516 28538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d9341e036b4684b00bc30c6150ef9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260619 02:38:43.390125 28528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:43.391173 28539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80d9341e036b4684b00bc30c6150ef9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d9341e036b4684b00bc30c6150ef9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } } }
I20260619 02:38:43.391294 28539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:43.391585 28540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80d9341e036b4684b00bc30c6150ef9c. Latest consensus state: current_term: 1 leader_uuid: "80d9341e036b4684b00bc30c6150ef9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d9341e036b4684b00bc30c6150ef9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } } }
I20260619 02:38:43.391664 28540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:43.391677 28547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:43.394938 28547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:43.400008 28547 catalog_manager.cc:1347] Generated new cluster ID: c8ef478474d6420fbd4298b71e8b1605
I20260619 02:38:43.400067 28547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:43.414649 28547 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:43.414971 28547 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:43.429066 28547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80d9341e036b4684b00bc30c6150ef9c: Generated new TSK 0
I20260619 02:38:43.429874 28547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:43.439837 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 28460
2026-06-19T02:38:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:43 dist-test-slave-9tkz krb5kdc[28559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:43 dist-test-slave-9tkz krb5kdc[28559](info): set up 2 sockets
Jun 19 02:38:43 dist-test-slave-9tkz krb5kdc[28559](info): commencing operation
krb5kdc: starting...
W20260619 02:38:44.474016 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:44 dist-test-slave-9tkz krb5kdc[28559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:44.596956 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:44.706786 28572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:44.707052 28572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:44.707132 28572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:44.711194 28572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:44.711252 28572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:44.711272 28572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:44.711288 28572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:44.711347 28572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:44.716478 28572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:44.717581 28572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:44.718627 28572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:44.727015 28578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:44.727068 28577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:44.727393 28572 server_base.cc:1034] running on GCE node
W20260619 02:38:44.727048 28580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:44.728052 28572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:44.729171 28572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:44.730324 28572 hybrid_clock.cc:648] HybridClock initialized: now 1781836724730294 us; error 41 us; skew 500 ppm
Jun 19 02:38:44 dist-test-slave-9tkz krb5kdc[28559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:44.738320 28572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:44.739337 28572 webserver.cc:460] Webserver started at http://127.0.0.1:42301/ using document root <none> and password file <none>
I20260619 02:38:44.739885 28572 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:44.739945 28572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:44.740116 28572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:44.742295 28572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "94261affb97541458f614c64da7c3243"
format_stamp: "Formatted at 2026-06-19 02:38:44 on dist-test-slave-9tkz"
server_key: "a05b945b3ac133107a892c909f08c041"
server_key_iv: "7ae42bb1f6e79500b66e8c088731e72c"
server_key_version: "encryptionkey@0"
I20260619 02:38:44.742836 28572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "94261affb97541458f614c64da7c3243"
format_stamp: "Formatted at 2026-06-19 02:38:44 on dist-test-slave-9tkz"
server_key: "a05b945b3ac133107a892c909f08c041"
server_key_iv: "7ae42bb1f6e79500b66e8c088731e72c"
server_key_version: "encryptionkey@0"
I20260619 02:38:44.746510 28572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:38:44.748906 28587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:44.750173 28572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:44.750316 28572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "94261affb97541458f614c64da7c3243"
format_stamp: "Formatted at 2026-06-19 02:38:44 on dist-test-slave-9tkz"
server_key: "a05b945b3ac133107a892c909f08c041"
server_key_iv: "7ae42bb1f6e79500b66e8c088731e72c"
server_key_version: "encryptionkey@0"
I20260619 02:38:44.750432 28572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:44.763803 28572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:44.767105 28572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:44.767315 28572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:44.774585 28572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36165
I20260619 02:38:44.774609 28639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36165 every 8 connection(s)
I20260619 02:38:44.775540 28572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:44.778285 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:44.783386 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 28572
I20260619 02:38:44.783564 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:44.783890 7947 external_mini_cluster.cc:1428] Setting key 8a71be7110eb193a50a306bab522ea6b
I20260619 02:38:44.784437 28640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243: Bootstrap starting.
I20260619 02:38:44.787230 28640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:44.788039 28640 log.cc:824] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:44.790182 28640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243: No bootstrap required, opened a new log
Jun 19 02:38:44 dist-test-slave-9tkz krb5kdc[28559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:38:44.793942 28640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94261affb97541458f614c64da7c3243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260619 02:38:44.794201 28640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:44.794281 28640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94261affb97541458f614c64da7c3243, State: Initialized, Role: FOLLOWER
I20260619 02:38:44.794821 28640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94261affb97541458f614c64da7c3243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260619 02:38:44.794987 28640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:44.795065 28640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:44.795174 28640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:44.796407 28640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94261affb97541458f614c64da7c3243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260619 02:38:44.796823 28640 leader_election.cc:302] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94261affb97541458f614c64da7c3243; no voters:
I20260619 02:38:44.797212 28640 leader_election.cc:288] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:44.797364 28645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:44.797650 28645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [term 1 LEADER]: Becoming Leader. State: Replica: 94261affb97541458f614c64da7c3243, State: Running, Role: LEADER
I20260619 02:38:44.798066 28645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94261affb97541458f614c64da7c3243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260619 02:38:44.798657 28640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:44.799602 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:44.785525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55759 (local address 127.0.0.1:36165)
0619 02:38:44.786075 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:44.786092 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:44.786140 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:38:44.786956 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:44.786977 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:44.787432 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:44.787802 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:44.788627 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:44.789751 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:44.790743 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:44.791238 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:44.791414 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:44.794032 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:44.794037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:44.794052 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:44.794081 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:44.796210 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:44.796666 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:44.796672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:44.796677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:44.796819 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:44.797130 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:44.797136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:44.797141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:44.797731 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:44.797930 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:44.798233 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:38:44.798551 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":172,"threads_started":1}
I20260619 02:38:44.799829 28646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94261affb97541458f614c64da7c3243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94261affb97541458f614c64da7c3243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } } }
I20260619 02:38:44.799948 28646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:44.800189 28646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94261affb97541458f614c64da7c3243. Latest consensus state: current_term: 1 leader_uuid: "94261affb97541458f614c64da7c3243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94261affb97541458f614c64da7c3243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } } }
I20260619 02:38:44.800262 28646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:44.800663 28654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:44.803220 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:44.804594 28654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:44.810328 28654 catalog_manager.cc:1347] Generated new cluster ID: 017b9055274442cc80488507790d4207
I20260619 02:38:44.810412 28654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:44.814687 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:44.805531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49333 (local address 127.0.0.1:36165)
0619 02:38:44.805667 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:44.805671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:44.805903 (+ 232us) server_negotiation.cc:407] Connection header received
0619 02:38:44.806052 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:44.806054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:44.806102 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:44.806200 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:44.807071 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:44.807869 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:44.809087 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:44.809449 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:44.809581 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:44.811286 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:44.811290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:44.811293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:44.811317 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:44.812942 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:44.813457 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:44.813461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:44.813463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:44.813564 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:44.813842 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:44.813845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:44.813846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:44.814038 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:44.814157 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:44.814408 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 02:38:44.814523 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:38:44.828207 28654 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:44.829221 28654 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:44.836031 28654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94261affb97541458f614c64da7c3243: Generated new TSK 0
I20260619 02:38:44.836795 28654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:44.846426 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 28572
2026-06-19T02:38:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:44 dist-test-slave-9tkz krb5kdc[28671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:44 dist-test-slave-9tkz krb5kdc[28671](info): set up 2 sockets
Jun 19 02:38:44 dist-test-slave-9tkz krb5kdc[28671](info): commencing operation
krb5kdc: starting...
W20260619 02:38:45.886018 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:45 dist-test-slave-9tkz krb5kdc[28671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:46.012825 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:46.126454 28684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:46.126726 28684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:46.126807 28684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:46.130981 28684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:46.131042 28684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:46.131062 28684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:46.131112 28684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:46.131143 28684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:46.136466 28684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:46.137643 28684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:46.138759 28684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:46.147598 28692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:46.147759 28689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:46.147680 28684 server_base.cc:1034] running on GCE node
W20260619 02:38:46.147616 28690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:46.148422 28684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:46.149534 28684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:46.150777 28684 hybrid_clock.cc:648] HybridClock initialized: now 1781836726150762 us; error 36 us; skew 500 ppm
Jun 19 02:38:46 dist-test-slave-9tkz krb5kdc[28671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:46.153472 28684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:46.154363 28684 webserver.cc:460] Webserver started at http://127.0.0.1:44015/ using document root <none> and password file <none>
I20260619 02:38:46.154944 28684 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:46.155042 28684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:46.155305 28684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:46.157346 28684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "f34d8502411f4fc09e9b49063887c977"
format_stamp: "Formatted at 2026-06-19 02:38:46 on dist-test-slave-9tkz"
server_key: "f8b1c4dcd1b150b7e66f9a7f59e06791"
server_key_iv: "75fea50fc694d4f7d0840c9ec4b14c65"
server_key_version: "encryptionkey@0"
I20260619 02:38:46.158005 28684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "f34d8502411f4fc09e9b49063887c977"
format_stamp: "Formatted at 2026-06-19 02:38:46 on dist-test-slave-9tkz"
server_key: "f8b1c4dcd1b150b7e66f9a7f59e06791"
server_key_iv: "75fea50fc694d4f7d0840c9ec4b14c65"
server_key_version: "encryptionkey@0"
I20260619 02:38:46.161834 28684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:38:46.164237 28699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:46.165385 28684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:46.165520 28684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "f34d8502411f4fc09e9b49063887c977"
format_stamp: "Formatted at 2026-06-19 02:38:46 on dist-test-slave-9tkz"
server_key: "f8b1c4dcd1b150b7e66f9a7f59e06791"
server_key_iv: "75fea50fc694d4f7d0840c9ec4b14c65"
server_key_version: "encryptionkey@0"
I20260619 02:38:46.165696 28684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:46.180806 28684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:46.183781 28684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:46.183987 28684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:46.191151 28684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33079
I20260619 02:38:46.191219 28751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33079 every 8 connection(s)
I20260619 02:38:46.192113 28684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:46.194870 28752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:46.199527 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 28684
I20260619 02:38:46.199681 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:46.199940 7947 external_mini_cluster.cc:1428] Setting key d29beef6fb9b7a9dcc45b05573ca4dbb
I20260619 02:38:46.200398 28752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977: Bootstrap starting.
I20260619 02:38:46.203227 28752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:46.204140 28752 log.cc:824] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:46.206332 28752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977: No bootstrap required, opened a new log
I20260619 02:38:46.209707 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:46.201352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53619 (local address 127.0.0.1:33079)
0619 02:38:46.201796 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:46.201829 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:46.201872 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:38:46.202799 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:46.202823 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:46.203262 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:46.203662 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:46.204087 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:46.205382 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:46.206583 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:46.207052 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:46.207171 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:46.208020 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:46.208025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:46.208037 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:46.208406 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:46.208472 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:46.208495 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:46.208568 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:46.208738 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 02:38:46.208949 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
I20260619 02:38:46.210376 28752 raft_consensus.cc:348] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34d8502411f4fc09e9b49063887c977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } }
I20260619 02:38:46.210638 28752 raft_consensus.cc:374] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:46.210703 28752 raft_consensus.cc:729] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34d8502411f4fc09e9b49063887c977, State: Initialized, Role: FOLLOWER
I20260619 02:38:46.211145 28752 consensus_queue.cc:260] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34d8502411f4fc09e9b49063887c977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } }
I20260619 02:38:46.211308 28752 raft_consensus.cc:388] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:46.211386 28752 raft_consensus.cc:482] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:46.211501 28752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:46.212529 28752 raft_consensus.cc:504] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34d8502411f4fc09e9b49063887c977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } }
I20260619 02:38:46.212682 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:46.212889 28752 leader_election.cc:302] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f34d8502411f4fc09e9b49063887c977; no voters:
I20260619 02:38:46.213191 28752 leader_election.cc:288] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:46.213300 28757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:46.213527 28757 raft_consensus.cc:686] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [term 1 LEADER]: Becoming Leader. State: Replica: f34d8502411f4fc09e9b49063887c977, State: Running, Role: LEADER
I20260619 02:38:46.213919 28757 consensus_queue.cc:237] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34d8502411f4fc09e9b49063887c977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } }
I20260619 02:38:46.214483 28752 sys_catalog.cc:564] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:46.215154 28757 sys_catalog.cc:455] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f34d8502411f4fc09e9b49063887c977. Latest consensus state: current_term: 1 leader_uuid: "f34d8502411f4fc09e9b49063887c977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34d8502411f4fc09e9b49063887c977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } } }
I20260619 02:38:46.215253 28757 sys_catalog.cc:458] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:46.215615 28758 sys_catalog.cc:455] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f34d8502411f4fc09e9b49063887c977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34d8502411f4fc09e9b49063887c977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } } }
I20260619 02:38:46.215724 28758 sys_catalog.cc:458] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:46.216694 28769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:46.219914 28769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:46.220920 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:46.214934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53891 (local address 127.0.0.1:33079)
0619 02:38:46.215024 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:46.215027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:46.216454 (+ 1427us) server_negotiation.cc:407] Connection header received
0619 02:38:46.216757 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:46.216760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:46.216805 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:46.216885 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:46.217224 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:46.218070 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:46.218978 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:46.219294 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:46.219390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:46.220328 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:46.220332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:46.220334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:46.220348 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:46.220375 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:46.220388 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:46.220451 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:46.220756 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 02:38:46.220805 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:38:46.225051 28769 catalog_manager.cc:1347] Generated new cluster ID: 8311e7ede5ab42868f8c738e03835308
I20260619 02:38:46.225108 28769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:46.235586 28769 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:46.236455 28769 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:46.241642 28769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f34d8502411f4fc09e9b49063887c977: Generated new TSK 0
I20260619 02:38:46.242256 28769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:46.246712 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 28684
2026-06-19T02:38:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:46 dist-test-slave-9tkz krb5kdc[28783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:46 dist-test-slave-9tkz krb5kdc[28783](info): set up 2 sockets
Jun 19 02:38:46 dist-test-slave-9tkz krb5kdc[28783](info): commencing operation
krb5kdc: starting...
W20260619 02:38:47.285909 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:47 dist-test-slave-9tkz krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:47.402148 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:47.517848 28796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:47.518157 28796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:47.518237 28796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:47.522324 28796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:47.522415 28796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:47.522455 28796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:47.522492 28796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:47.522531 28796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:47.527899 28796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:47.529003 28796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:47.530079 28796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:47.538323 28802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:47.538341 28801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:47.538341 28804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:47.538766 28796 server_base.cc:1034] running on GCE node
I20260619 02:38:47.539237 28796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:47.540261 28796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:47.541424 28796 hybrid_clock.cc:648] HybridClock initialized: now 1781836727541398 us; error 39 us; skew 500 ppm
Jun 19 02:38:47 dist-test-slave-9tkz krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:47.544080 28796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:47.544996 28796 webserver.cc:460] Webserver started at http://127.0.0.1:33961/ using document root <none> and password file <none>
I20260619 02:38:47.545619 28796 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:47.545708 28796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:47.545940 28796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:47.547940 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "e1aff28a919c44e89381852f8f44030f"
format_stamp: "Formatted at 2026-06-19 02:38:47 on dist-test-slave-9tkz"
server_key: "a79193e1606d8f16674193474d196eb5"
server_key_iv: "99486d8af04dc661f9b83d376dc6b95c"
server_key_version: "encryptionkey@0"
I20260619 02:38:47.548530 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "e1aff28a919c44e89381852f8f44030f"
format_stamp: "Formatted at 2026-06-19 02:38:47 on dist-test-slave-9tkz"
server_key: "a79193e1606d8f16674193474d196eb5"
server_key_iv: "99486d8af04dc661f9b83d376dc6b95c"
server_key_version: "encryptionkey@0"
I20260619 02:38:47.552155 28796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 02:38:47.554639 28811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:47.555789 28796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:47.555927 28796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "e1aff28a919c44e89381852f8f44030f"
format_stamp: "Formatted at 2026-06-19 02:38:47 on dist-test-slave-9tkz"
server_key: "a79193e1606d8f16674193474d196eb5"
server_key_iv: "99486d8af04dc661f9b83d376dc6b95c"
server_key_version: "encryptionkey@0"
I20260619 02:38:47.556043 28796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:47.560570 28796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:47.563441 28796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:47.563637 28796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:47.570585 28796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45983
I20260619 02:38:47.570611 28863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45983 every 8 connection(s)
I20260619 02:38:47.571470 28796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:47.574131 28864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:47.578347 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 28796
I20260619 02:38:47.578485 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:47.578768 7947 external_mini_cluster.cc:1428] Setting key 8dbbb9cb4a47a53c4d6bb96d6733449f
I20260619 02:38:47.579999 28864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f: Bootstrap starting.
I20260619 02:38:47.582417 28864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:47.583465 28864 log.cc:824] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:47.584419 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:47.580176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37441 (local address 127.0.0.1:45983)
0619 02:38:47.580605 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:47.580630 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:47.580684 (+ 54us) server_negotiation.cc:407] Connection header received
0619 02:38:47.581414 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:47.581432 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:47.581985 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:47.582348 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:47.583050 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:47.583054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:47.583066 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:47.583074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:47.583118 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:47.583136 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:47.583252 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:47.583372 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:38:47.583531 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260619 02:38:47.585457 28864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f: No bootstrap required, opened a new log
I20260619 02:38:47.587358 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:47.589516 28864 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aff28a919c44e89381852f8f44030f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20260619 02:38:47.589856 28864 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:47.589926 28864 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1aff28a919c44e89381852f8f44030f, State: Initialized, Role: FOLLOWER
I20260619 02:38:47.590353 28864 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aff28a919c44e89381852f8f44030f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20260619 02:38:47.590512 28864 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:47.590590 28864 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:47.590713 28864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:47.591117 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:47.589407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40227 (local address 127.0.0.1:45983)
0619 02:38:47.589528 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:47.589532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:47.589696 (+ 164us) server_negotiation.cc:407] Connection header received
0619 02:38:47.589769 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:47.589771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:47.589827 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:47.589933 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:47.590737 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:47.590741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:47.590743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:47.590755 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:47.590784 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:47.590799 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:47.590887 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:47.590958 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 02:38:47.590997 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:38:47.591900 28864 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aff28a919c44e89381852f8f44030f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20260619 02:38:47.592267 28864 leader_election.cc:302] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1aff28a919c44e89381852f8f44030f; no voters:
I20260619 02:38:47.592523 28864 leader_election.cc:288] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:47.592648 28874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:47.592888 28874 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [term 1 LEADER]: Becoming Leader. State: Replica: e1aff28a919c44e89381852f8f44030f, State: Running, Role: LEADER
I20260619 02:38:47.593190 28874 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aff28a919c44e89381852f8f44030f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20260619 02:38:47.593796 28864 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:47.595000 28876 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1aff28a919c44e89381852f8f44030f. Latest consensus state: current_term: 1 leader_uuid: "e1aff28a919c44e89381852f8f44030f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aff28a919c44e89381852f8f44030f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } } }
I20260619 02:38:47.595016 28875 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1aff28a919c44e89381852f8f44030f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aff28a919c44e89381852f8f44030f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } } }
I20260619 02:38:47.595146 28875 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:47.595145 28876 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:47.595620 28883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:47.598692 28883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:47.604164 28883 catalog_manager.cc:1347] Generated new cluster ID: 9b1e3338f27a4d26ac9dd13fb6602fc2
I20260619 02:38:47.604233 28883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:47.614301 28883 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:47.614641 28883 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:47.620245 28883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1aff28a919c44e89381852f8f44030f: Generated new TSK 0
I20260619 02:38:47.621033 28883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:47.628868 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 28796
2026-06-19T02:38:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:47 dist-test-slave-9tkz krb5kdc[28895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:47 dist-test-slave-9tkz krb5kdc[28895](info): set up 2 sockets
Jun 19 02:38:47 dist-test-slave-9tkz krb5kdc[28895](info): commencing operation
krb5kdc: starting...
W20260619 02:38:48.658004 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:48 dist-test-slave-9tkz krb5kdc[28895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:48.774286 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:48.883493 28908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:48.883818 28908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:48.883890 28908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:48.887907 28908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:48.888015 28908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:48.888054 28908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:48.888118 28908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:48.888156 28908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:48.893435 28908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:48.894524 28908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:48.895586 28908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:48.904017 28914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:48.904137 28908 server_base.cc:1034] running on GCE node
W20260619 02:38:48.904034 28916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:48.904039 28913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:48.904799 28908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:48.905946 28908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:48.907142 28908 hybrid_clock.cc:648] HybridClock initialized: now 1781836728907130 us; error 24 us; skew 500 ppm
Jun 19 02:38:48 dist-test-slave-9tkz krb5kdc[28895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:48.909821 28908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:48.910796 28908 webserver.cc:460] Webserver started at http://127.0.0.1:39215/ using document root <none> and password file <none>
I20260619 02:38:48.911353 28908 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:48.911442 28908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:48.911671 28908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:48.913698 28908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "06de1de0a8534fa6868270e73dbe4dca"
format_stamp: "Formatted at 2026-06-19 02:38:48 on dist-test-slave-9tkz"
server_key: "a1e0f0c35db17ba5b024ed26365f310c"
server_key_iv: "6d3f19a7e9a821af5729ed6735bd301a"
server_key_version: "encryptionkey@0"
I20260619 02:38:48.914273 28908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "06de1de0a8534fa6868270e73dbe4dca"
format_stamp: "Formatted at 2026-06-19 02:38:48 on dist-test-slave-9tkz"
server_key: "a1e0f0c35db17ba5b024ed26365f310c"
server_key_iv: "6d3f19a7e9a821af5729ed6735bd301a"
server_key_version: "encryptionkey@0"
I20260619 02:38:48.917897 28908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:38:48.920204 28923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:48.921363 28908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:48.921500 28908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "06de1de0a8534fa6868270e73dbe4dca"
format_stamp: "Formatted at 2026-06-19 02:38:48 on dist-test-slave-9tkz"
server_key: "a1e0f0c35db17ba5b024ed26365f310c"
server_key_iv: "6d3f19a7e9a821af5729ed6735bd301a"
server_key_version: "encryptionkey@0"
I20260619 02:38:48.921671 28908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:48.939364 28908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:48.942194 28908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:48.942413 28908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:48.949506 28908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36423
I20260619 02:38:48.949533 28975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36423 every 8 connection(s)
I20260619 02:38:48.950512 28908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:48.953220 28976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:48.958837 28976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca: Bootstrap starting.
I20260619 02:38:48.960137 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 28908
I20260619 02:38:48.960278 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:48.960578 7947 external_mini_cluster.cc:1428] Setting key 8bcadae9779b518f9a0ec70c1c751b26
I20260619 02:38:48.961701 28976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:48.962507 28976 log.cc:824] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:48.964295 28976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca: No bootstrap required, opened a new log
I20260619 02:38:48.967767 28976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06de1de0a8534fa6868270e73dbe4dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } }
I20260619 02:38:48.968006 28976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:48.968094 28976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06de1de0a8534fa6868270e73dbe4dca, State: Initialized, Role: FOLLOWER
I20260619 02:38:48.968560 28976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06de1de0a8534fa6868270e73dbe4dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } }
I20260619 02:38:48.968727 28976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:48.968804 28976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:48.968904 28976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [term 0 FOLLOWER]: Advancing to term 1
Jun 19 02:38:48 dist-test-slave-9tkz krb5kdc[28895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:38:48.970124 28976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06de1de0a8534fa6868270e73dbe4dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } }
I20260619 02:38:48.970479 28976 leader_election.cc:302] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06de1de0a8534fa6868270e73dbe4dca; no voters:
I20260619 02:38:48.970824 28976 leader_election.cc:288] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:48.970959 28981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:48.971246 28981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [term 1 LEADER]: Becoming Leader. State: Replica: 06de1de0a8534fa6868270e73dbe4dca, State: Running, Role: LEADER
I20260619 02:38:48.971609 28981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06de1de0a8534fa6868270e73dbe4dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } }
I20260619 02:38:48.972055 28976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:48.972834 28983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06de1de0a8534fa6868270e73dbe4dca. Latest consensus state: current_term: 1 leader_uuid: "06de1de0a8534fa6868270e73dbe4dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06de1de0a8534fa6868270e73dbe4dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } } }
I20260619 02:38:48.972982 28983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:48.973896 28982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06de1de0a8534fa6868270e73dbe4dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06de1de0a8534fa6868270e73dbe4dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } } }
I20260619 02:38:48.974067 28982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:48.975847 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:48.961766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:53311 (local address 127.0.0.1:36423)
0619 02:38:48.962201 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:48.962216 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:48.962251 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:38:48.962930 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:48.962949 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:48.963290 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:48.963628 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:48.964449 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:48.965448 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:48.966330 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:48.966811 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:48.966982 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:48.969974 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:48.969980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:48.969996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:48.970022 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:48.972105 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:48.972797 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:48.972804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:48.972811 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:48.973310 (+ 499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:48.973766 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:48.973771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:48.973776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:48.974316 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:48.974554 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:48.974958 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 02:38:48.975159 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":140,"threads_started":1}
W20260619 02:38:48.977221 28996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:38:48.977298 28996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:38:48.977370 28990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:48.978253 28990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:48.978793 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:48.983875 28990 catalog_manager.cc:1347] Generated new cluster ID: edca5373f8454289aa89a013a5eb3b7c
I20260619 02:38:48.983929 28990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:48.989214 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:48.980809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:36103 (local address 127.0.0.1:36423)
0619 02:38:48.980925 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:48.980929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:48.981179 (+ 250us) server_negotiation.cc:407] Connection header received
0619 02:38:48.981289 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:48.981292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:48.981335 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:48.981418 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:48.982223 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:48.982824 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:48.983680 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:48.984034 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:48.984138 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:48.985764 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:48.985768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:48.985771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:48.985793 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:48.987414 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:48.987945 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:48.987949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:48.987951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:48.988031 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:48.988377 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:48.988380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:48.988382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:48.988555 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:48.988675 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:48.988951 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 02:38:48.989064 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:38:48.994174 28990 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:48.995237 28990 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:49.001299 28990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06de1de0a8534fa6868270e73dbe4dca: Generated new TSK 0
I20260619 02:38:49.002125 28990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:49.014652 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 28908
W20260619 02:38:49.020082 28893 connection.cc:535] client connection to 127.0.0.1:36423 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36423): Connection reset by peer (error 104)
2026-06-19T02:38:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:49 dist-test-slave-9tkz krb5kdc[29007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:49 dist-test-slave-9tkz krb5kdc[29007](info): set up 2 sockets
Jun 19 02:38:49 dist-test-slave-9tkz krb5kdc[29007](info): commencing operation
krb5kdc: starting...
W20260619 02:38:50.037719 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:50 dist-test-slave-9tkz krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:50.154692 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:50.263324 29020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:50.263581 29020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:50.263621 29020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:50.267584 29020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:50.267647 29020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:50.267666 29020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:50.267683 29020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:50.267699 29020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:50.272835 29020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:50.273878 29020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:50.274892 29020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:50.283584 29026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:50.283573 29025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:50.283576 29028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:50.283826 29020 server_base.cc:1034] running on GCE node
I20260619 02:38:50.284552 29020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:50.285775 29020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:50.286976 29020 hybrid_clock.cc:648] HybridClock initialized: now 1781836730286969 us; error 35 us; skew 500 ppm
Jun 19 02:38:50 dist-test-slave-9tkz krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:50.294385 29020 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:50.295353 29020 webserver.cc:460] Webserver started at http://127.0.0.1:35097/ using document root <none> and password file <none>
I20260619 02:38:50.295954 29020 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:50.296056 29020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:50.296293 29020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:50.298501 29020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "c2607044af3a46f6903d8e83548cf558"
format_stamp: "Formatted at 2026-06-19 02:38:50 on dist-test-slave-9tkz"
server_key: "1b5f2001f301849df7721fb556b100eb"
server_key_iv: "fae7f4f4799b64aff9c2d6dd53e1b8c5"
server_key_version: "encryptionkey@0"
I20260619 02:38:50.299084 29020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "c2607044af3a46f6903d8e83548cf558"
format_stamp: "Formatted at 2026-06-19 02:38:50 on dist-test-slave-9tkz"
server_key: "1b5f2001f301849df7721fb556b100eb"
server_key_iv: "fae7f4f4799b64aff9c2d6dd53e1b8c5"
server_key_version: "encryptionkey@0"
I20260619 02:38:50.303072 29020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:38:50.305691 29035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:50.306886 29020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:38:50.307027 29020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "c2607044af3a46f6903d8e83548cf558"
format_stamp: "Formatted at 2026-06-19 02:38:50 on dist-test-slave-9tkz"
server_key: "1b5f2001f301849df7721fb556b100eb"
server_key_iv: "fae7f4f4799b64aff9c2d6dd53e1b8c5"
server_key_version: "encryptionkey@0"
I20260619 02:38:50.307164 29020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:50.321058 29020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:50.323815 29020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:50.324016 29020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:50.331426 29020 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45625
I20260619 02:38:50.331450 29087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45625 every 8 connection(s)
I20260619 02:38:50.332330 29020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:50.335115 29088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:50.340325 29088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558: Bootstrap starting.
I20260619 02:38:50.340618 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 29020
I20260619 02:38:50.340757 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:50.341048 7947 external_mini_cluster.cc:1428] Setting key 31750a2bd92baeb7dd58359f7c9b2ac1
I20260619 02:38:50.343024 29088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:50.343875 29088 log.cc:824] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:50.345747 29088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558: No bootstrap required, opened a new log
I20260619 02:38:50.349026 29088 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2607044af3a46f6903d8e83548cf558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } }
I20260619 02:38:50.349291 29088 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:50.349359 29088 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2607044af3a46f6903d8e83548cf558, State: Initialized, Role: FOLLOWER
I20260619 02:38:50.349864 29088 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2607044af3a46f6903d8e83548cf558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } }
I20260619 02:38:50.349982 29088 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:50.350067 29088 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:50.350178 29088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:50.350263 29091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:50.342404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:54125 (local address 127.0.0.1:45625)
0619 02:38:50.342815 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:50.342833 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:50.342875 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:38:50.343716 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:50.343740 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:50.344183 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:50.344533 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:50.344967 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:50.346008 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:50.346891 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:50.347358 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:50.347523 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:50.348390 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:50.348395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:50.348407 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:50.348778 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:50.348827 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:50.348847 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:50.348989 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:50.349143 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 02:38:50.349290 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260619 02:38:50.351050 29088 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2607044af3a46f6903d8e83548cf558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } }
I20260619 02:38:50.351408 29088 leader_election.cc:302] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2607044af3a46f6903d8e83548cf558; no voters:
I20260619 02:38:50.351742 29088 leader_election.cc:288] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:50.351828 29093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:50.352046 29093 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [term 1 LEADER]: Becoming Leader. State: Replica: c2607044af3a46f6903d8e83548cf558, State: Running, Role: LEADER
I20260619 02:38:50.352392 29093 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2607044af3a46f6903d8e83548cf558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } }
I20260619 02:38:50.352998 29088 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:50.353370 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:50.353974 29095 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2607044af3a46f6903d8e83548cf558. Latest consensus state: current_term: 1 leader_uuid: "c2607044af3a46f6903d8e83548cf558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2607044af3a46f6903d8e83548cf558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } } }
I20260619 02:38:50.354127 29095 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:50.354432 29094 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2607044af3a46f6903d8e83548cf558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2607044af3a46f6903d8e83548cf558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } } }
I20260619 02:38:50.354517 29094 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:50.355341 29102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:50.358290 29102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:50.360711 29091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:50.355768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:52367 (local address 127.0.0.1:45625)
0619 02:38:50.355872 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:50.355876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:50.356269 (+ 393us) server_negotiation.cc:407] Connection header received
0619 02:38:50.356358 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:50.356361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:50.356408 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:50.356501 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:50.356816 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:50.357585 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:50.358780 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:50.359105 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:50.359207 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:50.360201 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:50.360204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:50.360207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:50.360218 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:50.360244 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:50.360257 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:50.360343 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:50.360527 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 02:38:50.360559 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:38:50.363950 29102 catalog_manager.cc:1347] Generated new cluster ID: 1ebc34fc282f4f72b5cd8bc6258ca986
I20260619 02:38:50.364018 29102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:50.371197 29102 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:50.372021 29102 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:50.375764 29102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2607044af3a46f6903d8e83548cf558: Generated new TSK 0
I20260619 02:38:50.376556 29102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:50.386127 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 29020
W20260619 02:38:50.391346 29005 connection.cc:535] client connection to 127.0.0.1:45625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45625): Connection reset by peer (error 104)
2026-06-19T02:38:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:50 dist-test-slave-9tkz krb5kdc[29119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:50 dist-test-slave-9tkz krb5kdc[29119](info): set up 2 sockets
Jun 19 02:38:50 dist-test-slave-9tkz krb5kdc[29119](info): commencing operation
krb5kdc: starting...
W20260619 02:38:51.421862 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:51 dist-test-slave-9tkz krb5kdc[29119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:51.538698 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:51.649070 29132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:51.649345 29132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:51.649387 29132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:51.653262 29132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:51.653322 29132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:51.653340 29132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:51.653357 29132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:51.653370 29132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:51.658522 29132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:51.659540 29132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:51.660528 29132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:51.669016 29140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:51.669034 29137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:51.669034 29138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:51.669082 29132 server_base.cc:1034] running on GCE node
I20260619 02:38:51.669873 29132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:51.670985 29132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:51.672140 29132 hybrid_clock.cc:648] HybridClock initialized: now 1781836731672129 us; error 23 us; skew 500 ppm
Jun 19 02:38:51 dist-test-slave-9tkz krb5kdc[29119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:51.674885 29132 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:51.675743 29132 webserver.cc:460] Webserver started at http://127.0.0.1:40131/ using document root <none> and password file <none>
I20260619 02:38:51.676293 29132 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:51.676352 29132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:51.676607 29132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:51.678699 29132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "fd480e20c91f4db790c8bcc03ab248db"
format_stamp: "Formatted at 2026-06-19 02:38:51 on dist-test-slave-9tkz"
server_key: "e8cd96621036812aa1b1e7fd15ff4d71"
server_key_iv: "0686784fbe366335fcf0a9c276a75fc6"
server_key_version: "encryptionkey@0"
I20260619 02:38:51.679229 29132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "fd480e20c91f4db790c8bcc03ab248db"
format_stamp: "Formatted at 2026-06-19 02:38:51 on dist-test-slave-9tkz"
server_key: "e8cd96621036812aa1b1e7fd15ff4d71"
server_key_iv: "0686784fbe366335fcf0a9c276a75fc6"
server_key_version: "encryptionkey@0"
I20260619 02:38:51.682734 29132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:38:51.684985 29147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:51.686187 29132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:51.686307 29132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "fd480e20c91f4db790c8bcc03ab248db"
format_stamp: "Formatted at 2026-06-19 02:38:51 on dist-test-slave-9tkz"
server_key: "e8cd96621036812aa1b1e7fd15ff4d71"
server_key_iv: "0686784fbe366335fcf0a9c276a75fc6"
server_key_version: "encryptionkey@0"
I20260619 02:38:51.686403 29132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:51.690843 29132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:51.693454 29132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:51.693729 29132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:51.700546 29132 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34769
I20260619 02:38:51.700634 29199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34769 every 8 connection(s)
I20260619 02:38:51.701515 29132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:51.704447 29200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:51.704550 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 29132
I20260619 02:38:51.704663 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:51.705011 7947 external_mini_cluster.cc:1428] Setting key c2e7bc483a1cab008b9bcdd73fd5675b
I20260619 02:38:51.710457 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:51.706362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:37011 (local address 127.0.0.1:34769)
0619 02:38:51.706772 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:51.706792 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:51.706843 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:38:51.707550 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:51.707574 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:51.707953 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:51.708319 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:51.709080 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:51.709086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:51.709101 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:51.709112 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:51.709172 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:51.709196 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:51.709336 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:51.709407 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 02:38:51.709592 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260619 02:38:51.711998 29200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db: Bootstrap starting.
I20260619 02:38:51.713299 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:51.714650 29200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:51.715533 29200 log.cc:824] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:51.717535 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:51.715564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:39973 (local address 127.0.0.1:34769)
0619 02:38:51.715688 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:51.715692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:51.716135 (+ 443us) server_negotiation.cc:407] Connection header received
0619 02:38:51.716253 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:51.716258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:51.716289 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:51.716389 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:51.717120 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:51.717124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:51.717126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:51.717137 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:51.717161 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:51.717172 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:51.717262 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:51.717382 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:38:51.717421 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:38:51.717541 29200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db: No bootstrap required, opened a new log
I20260619 02:38:51.721266 29200 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd480e20c91f4db790c8bcc03ab248db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260619 02:38:51.721484 29200 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:51.721521 29200 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd480e20c91f4db790c8bcc03ab248db, State: Initialized, Role: FOLLOWER
I20260619 02:38:51.721967 29200 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd480e20c91f4db790c8bcc03ab248db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260619 02:38:51.722090 29200 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:51.722132 29200 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:51.722199 29200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:51.723212 29200 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd480e20c91f4db790c8bcc03ab248db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260619 02:38:51.723578 29200 leader_election.cc:302] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd480e20c91f4db790c8bcc03ab248db; no voters:
I20260619 02:38:51.723870 29200 leader_election.cc:288] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:51.724053 29210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:51.724345 29210 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [term 1 LEADER]: Becoming Leader. State: Replica: fd480e20c91f4db790c8bcc03ab248db, State: Running, Role: LEADER
I20260619 02:38:51.724670 29210 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd480e20c91f4db790c8bcc03ab248db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260619 02:38:51.725034 29200 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:51.725973 29210 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd480e20c91f4db790c8bcc03ab248db. Latest consensus state: current_term: 1 leader_uuid: "fd480e20c91f4db790c8bcc03ab248db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd480e20c91f4db790c8bcc03ab248db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } } }
I20260619 02:38:51.726097 29210 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:51.726382 29211 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd480e20c91f4db790c8bcc03ab248db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd480e20c91f4db790c8bcc03ab248db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } } }
I20260619 02:38:51.726467 29211 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:51.727813 29219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:51.730646 29219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:51.736109 29219 catalog_manager.cc:1347] Generated new cluster ID: 6ffa6bdbdf7846beb35813eaf01116bb
I20260619 02:38:51.736189 29219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:51.752172 29219 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:51.752513 29219 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:51.757148 29219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd480e20c91f4db790c8bcc03ab248db: Generated new TSK 0
I20260619 02:38:51.757899 29219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:51.765904 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 29132
2026-06-19T02:38:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:51 dist-test-slave-9tkz krb5kdc[29231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:51 dist-test-slave-9tkz krb5kdc[29231](info): set up 2 sockets
Jun 19 02:38:51 dist-test-slave-9tkz krb5kdc[29231](info): commencing operation
krb5kdc: starting...
W20260619 02:38:52.809711 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:52 dist-test-slave-9tkz krb5kdc[29231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:52.929991 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:53.037400 29244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:53.037708 29244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:53.037750 29244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:53.041618 29244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:53.041671 29244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:53.041689 29244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:53.041706 29244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:53.041721 29244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:53.046809 29244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:53.047788 29244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:53.048751 29244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:53.057163 29252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:53.057204 29250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:53.057291 29244 server_base.cc:1034] running on GCE node
W20260619 02:38:53.057207 29249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:53.058063 29244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:53.059161 29244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:53.060345 29244 hybrid_clock.cc:648] HybridClock initialized: now 1781836733060327 us; error 30 us; skew 500 ppm
Jun 19 02:38:53 dist-test-slave-9tkz krb5kdc[29231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:53.063187 29244 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:53.063989 29244 webserver.cc:460] Webserver started at http://127.0.0.1:35141/ using document root <none> and password file <none>
I20260619 02:38:53.064489 29244 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:53.064544 29244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:53.064718 29244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:53.066798 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "3593c959121e43e99006298c323125e9"
format_stamp: "Formatted at 2026-06-19 02:38:53 on dist-test-slave-9tkz"
server_key: "d2804789910c0a63ab83a255363c3da3"
server_key_iv: "f29a9b4345cee1430d076635f0fb90c8"
server_key_version: "encryptionkey@0"
I20260619 02:38:53.067306 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "3593c959121e43e99006298c323125e9"
format_stamp: "Formatted at 2026-06-19 02:38:53 on dist-test-slave-9tkz"
server_key: "d2804789910c0a63ab83a255363c3da3"
server_key_iv: "f29a9b4345cee1430d076635f0fb90c8"
server_key_version: "encryptionkey@0"
I20260619 02:38:53.070737 29244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:38:53.072979 29259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:53.074200 29244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:53.074319 29244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "3593c959121e43e99006298c323125e9"
format_stamp: "Formatted at 2026-06-19 02:38:53 on dist-test-slave-9tkz"
server_key: "d2804789910c0a63ab83a255363c3da3"
server_key_iv: "f29a9b4345cee1430d076635f0fb90c8"
server_key_version: "encryptionkey@0"
I20260619 02:38:53.074414 29244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:53.095340 29244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:53.098664 29244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:53.098870 29244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:53.105624 29244 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35135
I20260619 02:38:53.105681 29311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35135 every 8 connection(s)
I20260619 02:38:53.106534 29244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:53.109225 29312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:53.114594 29312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9: Bootstrap starting.
I20260619 02:38:53.115556 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 29244
I20260619 02:38:53.115703 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:53.116022 7947 external_mini_cluster.cc:1428] Setting key f8aa6da3bb26204981a9887f1c161789
I20260619 02:38:53.117627 29312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:53.118342 29312 log.cc:824] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:53.120301 29312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9: No bootstrap required, opened a new log
I20260619 02:38:53.123677 29312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593c959121e43e99006298c323125e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260619 02:38:53.123888 29312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:53.123920 29312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3593c959121e43e99006298c323125e9, State: Initialized, Role: FOLLOWER
I20260619 02:38:53.124421 29312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593c959121e43e99006298c323125e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
Jun 19 02:38:53 dist-test-slave-9tkz krb5kdc[29231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:38:53.124570 29312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:53.124665 29312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:53.124765 29312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:53.125774 29312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593c959121e43e99006298c323125e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260619 02:38:53.126096 29312 leader_election.cc:302] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3593c959121e43e99006298c323125e9; no voters:
I20260619 02:38:53.126372 29312 leader_election.cc:288] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:53.126495 29317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:53.126695 29317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [term 1 LEADER]: Becoming Leader. State: Replica: 3593c959121e43e99006298c323125e9, State: Running, Role: LEADER
I20260619 02:38:53.127072 29317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593c959121e43e99006298c323125e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260619 02:38:53.127658 29312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:53.128808 29318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3593c959121e43e99006298c323125e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593c959121e43e99006298c323125e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } } }
I20260619 02:38:53.128918 29318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:53.129189 29319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3593c959121e43e99006298c323125e9. Latest consensus state: current_term: 1 leader_uuid: "3593c959121e43e99006298c323125e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593c959121e43e99006298c323125e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } } }
I20260619 02:38:53.129354 29319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:53.131297 29315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:53.117196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:53805 (local address 127.0.0.1:35135)
0619 02:38:53.117603 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:53.117619 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:53.117652 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:38:53.118333 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:53.118350 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:53.118745 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:53.119087 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:53.119989 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:53.121006 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:53.121927 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:53.122394 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:53.122566 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:53.125221 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:53.125227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:53.125240 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:53.125265 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:53.127339 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:53.127947 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:53.127954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:53.127959 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:53.128096 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:53.128537 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:53.128542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:53.128546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:53.129114 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:53.129327 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:53.130151 (+ 824us) server_negotiation.cc:299] Negotiation successful
0619 02:38:53.130365 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":118,"threads_started":1}
W20260619 02:38:53.132627 29332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:38:53.132707 29332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:38:53.132766 29326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:53.133935 29326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:53.134320 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:53.139266 29326 catalog_manager.cc:1347] Generated new cluster ID: 54ba31d23bca49b39a332b1f574c883e
I20260619 02:38:53.139371 29326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:53.146445 29315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:53.136509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:36391 (local address 127.0.0.1:35135)
0619 02:38:53.136604 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:53.136609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:53.136838 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:38:53.136940 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:53.136943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:53.136989 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:53.137069 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:53.137959 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:53.139147 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:53.140315 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:53.140704 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:53.140801 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:53.142596 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:53.142600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:53.142603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:53.142627 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:53.144157 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:53.144730 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:53.144734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:53.144735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:53.144816 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:53.145208 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:53.145212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:53.145214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:53.145419 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:53.145589 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:53.146205 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 02:38:53.146292 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:38:53.155210 29326 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:53.156118 29326 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:53.164695 29326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3593c959121e43e99006298c323125e9: Generated new TSK 0
I20260619 02:38:53.165517 29326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:53.172420 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 29244
W20260619 02:38:53.177932 29229 connection.cc:535] client connection to 127.0.0.1:35135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35135): Connection reset by peer (error 104)
2026-06-19T02:38:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:53 dist-test-slave-9tkz krb5kdc[29343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:53 dist-test-slave-9tkz krb5kdc[29343](info): set up 2 sockets
Jun 19 02:38:53 dist-test-slave-9tkz krb5kdc[29343](info): commencing operation
krb5kdc: starting...
W20260619 02:38:54.205909 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:54 dist-test-slave-9tkz krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:54.327941 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:54.438484 29356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:54.438891 29356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:54.439023 29356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:54.443087 29356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:54.443145 29356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:54.443166 29356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:54.443182 29356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:54.443198 29356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:54.448423 29356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:54.449429 29356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:54.450479 29356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:54.458626 29362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:54.458619 29364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:54.458648 29361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:54.458956 29356 server_base.cc:1034] running on GCE node
I20260619 02:38:54.459492 29356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:54.460546 29356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:54.461732 29356 hybrid_clock.cc:648] HybridClock initialized: now 1781836734461713 us; error 30 us; skew 500 ppm
Jun 19 02:38:54 dist-test-slave-9tkz krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:54.464332 29356 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:54.465224 29356 webserver.cc:460] Webserver started at http://127.0.0.1:34121/ using document root <none> and password file <none>
I20260619 02:38:54.465871 29356 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:54.465930 29356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:54.466176 29356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:54.468211 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "50ed2f0e849444598076dc29e2b7b785"
format_stamp: "Formatted at 2026-06-19 02:38:54 on dist-test-slave-9tkz"
server_key: "f31d58b0d9283227742076acb00bfd1a"
server_key_iv: "f30dad3267075da53911bd13cef63098"
server_key_version: "encryptionkey@0"
I20260619 02:38:54.468767 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "50ed2f0e849444598076dc29e2b7b785"
format_stamp: "Formatted at 2026-06-19 02:38:54 on dist-test-slave-9tkz"
server_key: "f31d58b0d9283227742076acb00bfd1a"
server_key_iv: "f30dad3267075da53911bd13cef63098"
server_key_version: "encryptionkey@0"
I20260619 02:38:54.472395 29356 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 02:38:54.474786 29371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:54.475979 29356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:38:54.476110 29356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "50ed2f0e849444598076dc29e2b7b785"
format_stamp: "Formatted at 2026-06-19 02:38:54 on dist-test-slave-9tkz"
server_key: "f31d58b0d9283227742076acb00bfd1a"
server_key_iv: "f30dad3267075da53911bd13cef63098"
server_key_version: "encryptionkey@0"
I20260619 02:38:54.476214 29356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:54.494918 29356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:54.497771 29356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:54.497985 29356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:54.505183 29356 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34053
I20260619 02:38:54.505209 29423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34053 every 8 connection(s)
I20260619 02:38:54.506213 29356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:54.508966 29424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:54.513934 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 29356
I20260619 02:38:54.514113 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:54.514413 7947 external_mini_cluster.cc:1428] Setting key d937729af302180d5e0a5c869a21d730
I20260619 02:38:54.514876 29424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785: Bootstrap starting.
I20260619 02:38:54.517786 29424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:54.518579 29424 log.cc:824] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:54.520568 29424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785: No bootstrap required, opened a new log
I20260619 02:38:54.524016 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:54.515919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:45581 (local address 127.0.0.1:34053)
0619 02:38:54.516385 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:54.516400 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:54.516445 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:38:54.517134 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:54.517158 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:54.517521 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:54.517960 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:54.518388 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:54.519654 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:54.520591 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:54.521123 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:54.521311 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:54.522222 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:54.522227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:54.522240 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:54.522752 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:54.522809 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:54.522832 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:54.522974 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:54.523173 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 02:38:54.523307 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":154,"threads_started":1}
I20260619 02:38:54.524739 29424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ed2f0e849444598076dc29e2b7b785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260619 02:38:54.525025 29424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:54.525098 29424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ed2f0e849444598076dc29e2b7b785, State: Initialized, Role: FOLLOWER
I20260619 02:38:54.525508 29424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ed2f0e849444598076dc29e2b7b785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260619 02:38:54.525735 29424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:54.525804 29424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:54.525918 29424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:54.527004 29424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ed2f0e849444598076dc29e2b7b785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260619 02:38:54.527227 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:54.527362 29424 leader_election.cc:302] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50ed2f0e849444598076dc29e2b7b785; no voters:
I20260619 02:38:54.527691 29424 leader_election.cc:288] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:54.527776 29429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:54.528041 29429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [term 1 LEADER]: Becoming Leader. State: Replica: 50ed2f0e849444598076dc29e2b7b785, State: Running, Role: LEADER
I20260619 02:38:54.528349 29429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ed2f0e849444598076dc29e2b7b785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260619 02:38:54.529079 29424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:54.529469 29429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50ed2f0e849444598076dc29e2b7b785. Latest consensus state: current_term: 1 leader_uuid: "50ed2f0e849444598076dc29e2b7b785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ed2f0e849444598076dc29e2b7b785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } } }
I20260619 02:38:54.529618 29429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:54.529925 29430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50ed2f0e849444598076dc29e2b7b785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ed2f0e849444598076dc29e2b7b785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } } }
I20260619 02:38:54.530001 29430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:54.530046 29437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:54.533185 29437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:54.536046 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:54.529523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:51493 (local address 127.0.0.1:34053)
0619 02:38:54.529656 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:54.529660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:54.531427 (+ 1767us) server_negotiation.cc:407] Connection header received
0619 02:38:54.531615 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:54.531619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:54.531666 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:54.531715 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:54.532153 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:54.532889 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:54.534113 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:54.534426 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:54.534541 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:54.535531 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:54.535535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:54.535538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:54.535551 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:54.535579 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:54.535599 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:54.535706 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:54.535862 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 02:38:54.535896 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:38:54.539762 29437 catalog_manager.cc:1347] Generated new cluster ID: ea6fd6e29c504ae79db4dd453276c97a
I20260619 02:38:54.539844 29437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:54.546279 29437 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:54.547561 29437 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:54.552917 29437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50ed2f0e849444598076dc29e2b7b785: Generated new TSK 0
I20260619 02:38:54.553705 29437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:54.557534 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 29356
W20260619 02:38:54.562837 29341 connection.cc:535] client connection to 127.0.0.1:34053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34053): Connection reset by peer (error 104)
2026-06-19T02:38:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:54 dist-test-slave-9tkz krb5kdc[29455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:54 dist-test-slave-9tkz krb5kdc[29455](info): set up 2 sockets
Jun 19 02:38:54 dist-test-slave-9tkz krb5kdc[29455](info): commencing operation
krb5kdc: starting...
W20260619 02:38:55.593981 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:55 dist-test-slave-9tkz krb5kdc[29455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:38:55.710220 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:55.818632 29468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:55.818980 29468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:55.819060 29468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:55.823167 29468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:55.823262 29468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:55.823302 29468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:55.823354 29468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:55.823392 29468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:55.828894 29468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:55.829973 29468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:55.830968 29468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:55.839547 29473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:55.839534 29476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:55.839546 29474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:55.839582 29468 server_base.cc:1034] running on GCE node
I20260619 02:38:55.840354 29468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:55.841437 29468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:55.842597 29468 hybrid_clock.cc:648] HybridClock initialized: now 1781836735842583 us; error 25 us; skew 500 ppm
Jun 19 02:38:55 dist-test-slave-9tkz krb5kdc[29455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:55.849582 29468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:38:55.850498 29468 webserver.cc:460] Webserver started at http://127.0.0.1:33435/ using document root <none> and password file <none>
I20260619 02:38:55.851075 29468 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:55.851166 29468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:55.851468 29468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:55.853499 29468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "37e61ef9eb46452f9206098fe4adf971"
format_stamp: "Formatted at 2026-06-19 02:38:55 on dist-test-slave-9tkz"
server_key: "06ab128c37f178183a34a8659998c587"
server_key_iv: "914c28d869592fd5ee560c5fd4e3d19a"
server_key_version: "encryptionkey@0"
I20260619 02:38:55.854118 29468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "37e61ef9eb46452f9206098fe4adf971"
format_stamp: "Formatted at 2026-06-19 02:38:55 on dist-test-slave-9tkz"
server_key: "06ab128c37f178183a34a8659998c587"
server_key_iv: "914c28d869592fd5ee560c5fd4e3d19a"
server_key_version: "encryptionkey@0"
I20260619 02:38:55.857810 29468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:38:55.860200 29483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:55.861445 29468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:38:55.861634 29468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "37e61ef9eb46452f9206098fe4adf971"
format_stamp: "Formatted at 2026-06-19 02:38:55 on dist-test-slave-9tkz"
server_key: "06ab128c37f178183a34a8659998c587"
server_key_iv: "914c28d869592fd5ee560c5fd4e3d19a"
server_key_version: "encryptionkey@0"
I20260619 02:38:55.861752 29468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:55.866333 29468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:55.869097 29468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:55.869287 29468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:55.876060 29468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35297
I20260619 02:38:55.876092 29535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35297 every 8 connection(s)
I20260619 02:38:55.877022 29468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:55.879660 29536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:55.885076 29536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971: Bootstrap starting.
I20260619 02:38:55.886344 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 29468
I20260619 02:38:55.886514 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:55.886839 7947 external_mini_cluster.cc:1428] Setting key 2c8138a61ddb5232101e824fb3b2efad
I20260619 02:38:55.887969 29536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:55.888749 29536 log.cc:824] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:55.890872 29536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971: No bootstrap required, opened a new log
I20260619 02:38:55.892481 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:55.888024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:48119 (local address 127.0.0.1:35297)
0619 02:38:55.888495 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:55.888511 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:55.888547 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:38:55.889461 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:55.889487 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:55.889850 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:55.890244 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:55.891071 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:55.891077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:55.891092 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:55.891104 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:55.891169 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:55.891196 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:55.891315 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:55.891427 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 02:38:55.891579 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260619 02:38:55.895135 29536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e61ef9eb46452f9206098fe4adf971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260619 02:38:55.895375 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:55.895398 29536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:55.895506 29536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e61ef9eb46452f9206098fe4adf971, State: Initialized, Role: FOLLOWER
I20260619 02:38:55.895980 29536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e61ef9eb46452f9206098fe4adf971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260619 02:38:55.896106 29536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:55.896188 29536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:55.896306 29536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:55.897276 29536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e61ef9eb46452f9206098fe4adf971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260619 02:38:55.897668 29536 leader_election.cc:302] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e61ef9eb46452f9206098fe4adf971; no voters:
I20260619 02:38:55.898015 29536 leader_election.cc:288] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:55.898079 29546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:55.898313 29546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [term 1 LEADER]: Becoming Leader. State: Replica: 37e61ef9eb46452f9206098fe4adf971, State: Running, Role: LEADER
I20260619 02:38:55.898660 29546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e61ef9eb46452f9206098fe4adf971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260619 02:38:55.899425 29536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:55.899727 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:55.897805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:50789 (local address 127.0.0.1:35297)
0619 02:38:55.897924 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:55.897930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:55.898120 (+ 190us) server_negotiation.cc:407] Connection header received
0619 02:38:55.898319 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:55.898323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:55.898359 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:55.898446 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:55.899276 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:55.899279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:55.899282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:55.899293 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:55.899319 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:55.899331 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:55.899385 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:55.899577 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:38:55.899618 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:38:55.899828 29546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37e61ef9eb46452f9206098fe4adf971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e61ef9eb46452f9206098fe4adf971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } } }
I20260619 02:38:55.899978 29546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:55.900215 29546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37e61ef9eb46452f9206098fe4adf971. Latest consensus state: current_term: 1 leader_uuid: "37e61ef9eb46452f9206098fe4adf971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e61ef9eb46452f9206098fe4adf971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } } }
I20260619 02:38:55.900332 29546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971 [sys.catalog]: This master's current role is: LEADER
W20260619 02:38:55.904170 29561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:38:55.904232 29561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:38:55.904335 29550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:55.904995 29550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:55.910291 29550 catalog_manager.cc:1347] Generated new cluster ID: 975abf07354043c995470674047d04f6
I20260619 02:38:55.910370 29550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:55.923076 29550 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:55.923450 29550 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:55.927784 29550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37e61ef9eb46452f9206098fe4adf971: Generated new TSK 0
I20260619 02:38:55.928452 29550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:55.937458 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 29468
2026-06-19T02:38:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:55 dist-test-slave-9tkz krb5kdc[29567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:55 dist-test-slave-9tkz krb5kdc[29567](info): set up 2 sockets
Jun 19 02:38:55 dist-test-slave-9tkz krb5kdc[29567](info): commencing operation
krb5kdc: starting...
W20260619 02:38:56.983032 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:57 dist-test-slave-9tkz krb5kdc[29567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:38:57.112278 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44451
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:57.221907 29580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:57.222231 29580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:57.222321 29580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:57.226671 29580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:57.226866 29580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:57.226912 29580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:57.226971 29580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:57.227008 29580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:57.232776 29580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:44451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:57.233968 29580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:57.235062 29580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:57.243891 29585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:57.243882 29586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:57.243882 29588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:57.244316 29580 server_base.cc:1034] running on GCE node
I20260619 02:38:57.244915 29580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:57.246220 29580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:57.247424 29580 hybrid_clock.cc:648] HybridClock initialized: now 1781836737247411 us; error 26 us; skew 500 ppm
Jun 19 02:38:57 dist-test-slave-9tkz krb5kdc[29567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836737, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:57.250269 29580 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:38:57.251189 29580 webserver.cc:460] Webserver started at http://127.7.194.254:37923/ using document root <none> and password file <none>
I20260619 02:38:57.251780 29580 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:57.251869 29580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:57.252095 29580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:57.254299 29580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "7dae6ec5dd9a482ba04470497e1187c6"
format_stamp: "Formatted at 2026-06-19 02:38:57 on dist-test-slave-9tkz"
server_key: "94f9374af5044e0aa81a8bd3fdf59740"
server_key_iv: "d039f1ad374df80bcd4ecd39e2308040"
server_key_version: "encryptionkey@0"
I20260619 02:38:57.254875 29580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "7dae6ec5dd9a482ba04470497e1187c6"
format_stamp: "Formatted at 2026-06-19 02:38:57 on dist-test-slave-9tkz"
server_key: "94f9374af5044e0aa81a8bd3fdf59740"
server_key_iv: "d039f1ad374df80bcd4ecd39e2308040"
server_key_version: "encryptionkey@0"
I20260619 02:38:57.258481 29580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:38:57.260794 29595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:57.262038 29580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:38:57.262171 29580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "7dae6ec5dd9a482ba04470497e1187c6"
format_stamp: "Formatted at 2026-06-19 02:38:57 on dist-test-slave-9tkz"
server_key: "94f9374af5044e0aa81a8bd3fdf59740"
server_key_iv: "d039f1ad374df80bcd4ecd39e2308040"
server_key_version: "encryptionkey@0"
I20260619 02:38:57.262280 29580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:57.280465 29580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:57.283767 29580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:57.283980 29580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:57.291163 29580 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:44451
I20260619 02:38:57.291185 29647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:44451 every 8 connection(s)
I20260619 02:38:57.292121 29580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:57.294937 29648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:57.298381 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 29580
I20260619 02:38:57.298560 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:57.298858 7947 external_mini_cluster.cc:1428] Setting key bed31d60df2e64208230a1f9d7dfbd6a
I20260619 02:38:57.301169 29648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6: Bootstrap starting.
I20260619 02:38:57.303423 29648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:57.304154 29648 log.cc:824] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:57.306079 29648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6: No bootstrap required, opened a new log
Jun 19 02:38:57 dist-test-slave-9tkz krb5kdc[29567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:38:57.310566 29648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dae6ec5dd9a482ba04470497e1187c6" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44451 } }
I20260619 02:38:57.310843 29648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:57.310907 29648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dae6ec5dd9a482ba04470497e1187c6, State: Initialized, Role: FOLLOWER
I20260619 02:38:57.311403 29648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dae6ec5dd9a482ba04470497e1187c6" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44451 } }
I20260619 02:38:57.311573 29648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:57.311658 29648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:57.311791 29648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:57.312896 29648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dae6ec5dd9a482ba04470497e1187c6" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44451 } }
I20260619 02:38:57.313225 29648 leader_election.cc:302] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dae6ec5dd9a482ba04470497e1187c6; no voters:
I20260619 02:38:57.313575 29648 leader_election.cc:288] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:57.313668 29653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:57.313832 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:57.300329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41579 (local address 127.7.194.254:44451)
0619 02:38:57.300776 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:57.300791 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:57.300826 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:38:57.301521 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:57.301595 (+ 74us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:57.301953 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:57.302281 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:57.303146 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:57.304177 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:57.305273 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:57.305800 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:57.305978 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:57.308723 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:57.308728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:57.308741 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:57.308769 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:57.310596 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:57.311167 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:57.311174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:57.311179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:57.311311 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:57.311676 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:57.311679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:57.311680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:57.312132 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:57.312282 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:57.312619 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 02:38:57.312857 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
I20260619 02:38:57.313920 29653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [term 1 LEADER]: Becoming Leader. State: Replica: 7dae6ec5dd9a482ba04470497e1187c6, State: Running, Role: LEADER
I20260619 02:38:57.314218 29653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dae6ec5dd9a482ba04470497e1187c6" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44451 } }
I20260619 02:38:57.314941 29648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:57.315430 29655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dae6ec5dd9a482ba04470497e1187c6. Latest consensus state: current_term: 1 leader_uuid: "7dae6ec5dd9a482ba04470497e1187c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dae6ec5dd9a482ba04470497e1187c6" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44451 } } }
I20260619 02:38:57.315539 29655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:57.316159 29654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dae6ec5dd9a482ba04470497e1187c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dae6ec5dd9a482ba04470497e1187c6" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 44451 } } }
I20260619 02:38:57.316248 29654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:57.316488 29658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:57.317351 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:57.320245 29658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:57.326444 29658 catalog_manager.cc:1347] Generated new cluster ID: c08883bb62304c0393b15974f55fa3cc
I20260619 02:38:57.326562 29658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:57.328262 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:57.319711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55885 (local address 127.7.194.254:44451)
0619 02:38:57.319824 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:57.319837 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:57.320037 (+ 200us) server_negotiation.cc:407] Connection header received
0619 02:38:57.320155 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:57.320158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:57.320204 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:57.320291 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:57.321111 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:57.321949 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:57.322790 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:57.323051 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:57.323175 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:57.324705 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:57.324709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:57.324711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:38:57.324734 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:57.326489 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:57.327008 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:57.327010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:57.327012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:57.327078 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:38:57.327403 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:38:57.327408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:38:57.327410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:38:57.327584 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:57.327737 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:57.328000 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 02:38:57.328093 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:38:57.355702 29658 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:57.356627 29658 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:57.372189 29658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dae6ec5dd9a482ba04470497e1187c6: Generated new TSK 0
I20260619 02:38:57.372977 29658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:57.382238 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 29580
W20260619 02:38:57.387719 29565 connection.cc:535] client connection to 127.7.194.254:44451 recv error: Network error: failed to read from TLS socket (remote: 127.7.194.254:44451): Connection reset by peer (error 104)
2026-06-19T02:38:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:57 dist-test-slave-9tkz krb5kdc[29679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:57 dist-test-slave-9tkz krb5kdc[29679](info): set up 2 sockets
Jun 19 02:38:57 dist-test-slave-9tkz krb5kdc[29679](info): commencing operation
krb5kdc: starting...
W20260619 02:38:58.417809 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:58 dist-test-slave-9tkz krb5kdc[29679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:38:58.538280 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:39357
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:39357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:38:58.650300 29692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:38:58.650601 29692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:38:58.650681 29692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:38:58.654769 29692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:38:58.654860 29692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:38:58.654908 29692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:38:58.654968 29692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:38:58.655006 29692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:38:58.660197 29692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:39357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:39357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:38:58.661258 29692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:38:58.662295 29692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:38:58.670425 29698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:58.670425 29700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:38:58.670487 29697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:38:58.670627 29692 server_base.cc:1034] running on GCE node
I20260619 02:38:58.671196 29692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:38:58.672289 29692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:38:58.673476 29692 hybrid_clock.cc:648] HybridClock initialized: now 1781836738673460 us; error 29 us; skew 500 ppm
Jun 19 02:38:58 dist-test-slave-9tkz krb5kdc[29679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836738, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:38:58.676219 29692 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:38:58.677078 29692 webserver.cc:460] Webserver started at http://127.7.194.254:41939/ using document root <none> and password file <none>
I20260619 02:38:58.677824 29692 fs_manager.cc:359] Metadata directory not provided
I20260619 02:38:58.677894 29692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:38:58.678125 29692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:38:58.680123 29692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "8062efcd52ff4b5cae783fa1441c2b8f"
format_stamp: "Formatted at 2026-06-19 02:38:58 on dist-test-slave-9tkz"
server_key: "bdd3ef03739aa0d3281645a81db83636"
server_key_iv: "3282d638d6b3a11a8ce099517e5a7dd8"
server_key_version: "encryptionkey@0"
I20260619 02:38:58.680682 29692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "8062efcd52ff4b5cae783fa1441c2b8f"
format_stamp: "Formatted at 2026-06-19 02:38:58 on dist-test-slave-9tkz"
server_key: "bdd3ef03739aa0d3281645a81db83636"
server_key_iv: "3282d638d6b3a11a8ce099517e5a7dd8"
server_key_version: "encryptionkey@0"
I20260619 02:38:58.684294 29692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 02:38:58.686769 29707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:38:58.688006 29692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:38:58.688138 29692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "8062efcd52ff4b5cae783fa1441c2b8f"
format_stamp: "Formatted at 2026-06-19 02:38:58 on dist-test-slave-9tkz"
server_key: "bdd3ef03739aa0d3281645a81db83636"
server_key_iv: "3282d638d6b3a11a8ce099517e5a7dd8"
server_key_version: "encryptionkey@0"
I20260619 02:38:58.688237 29692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:38:58.724786 29692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:38:58.728297 29692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:38:58.728518 29692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:38:58.735500 29692 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:39357
I20260619 02:38:58.735544 29759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:39357 every 8 connection(s)
I20260619 02:38:58.736526 29692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:38:58.739351 29760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:38:58.744904 29760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f: Bootstrap starting.
I20260619 02:38:58.744915 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 29692
I20260619 02:38:58.745020 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:38:58.745296 7947 external_mini_cluster.cc:1428] Setting key 97f9c52959b08af9023c6f8237921c1c
I20260619 02:38:58.747457 29760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f: Neither blocks nor log segments found. Creating new log.
I20260619 02:38:58.748169 29760 log.cc:824] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:38:58.750075 29760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f: No bootstrap required, opened a new log
I20260619 02:38:58.753455 29760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8062efcd52ff4b5cae783fa1441c2b8f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39357 } }
I20260619 02:38:58.753773 29760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:38:58.753885 29760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8062efcd52ff4b5cae783fa1441c2b8f, State: Initialized, Role: FOLLOWER
I20260619 02:38:58.754321 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:58.746658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45293 (local address 127.7.194.254:39357)
0619 02:38:58.747085 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:58.747100 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:58.747136 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:38:58.747784 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:58.747821 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:58.748165 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:58.748535 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:58.748964 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:58.749969 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:58.750932 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:58.751434 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:58.751614 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:58.752516 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:58.752521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:58.752533 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:58.752897 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:58.752948 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:58.752969 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:58.753107 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:58.753298 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 02:38:58.753437 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":142,"threads_started":1}
I20260619 02:38:58.754359 29760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8062efcd52ff4b5cae783fa1441c2b8f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39357 } }
I20260619 02:38:58.754557 29760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:38:58.754616 29760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:38:58.754700 29760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:38:58.755735 29760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8062efcd52ff4b5cae783fa1441c2b8f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39357 } }
I20260619 02:38:58.756062 29760 leader_election.cc:302] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8062efcd52ff4b5cae783fa1441c2b8f; no voters:
I20260619 02:38:58.756417 29760 leader_election.cc:288] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:38:58.756512 29765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:38:58.756738 29765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [term 1 LEADER]: Becoming Leader. State: Replica: 8062efcd52ff4b5cae783fa1441c2b8f, State: Running, Role: LEADER
I20260619 02:38:58.757052 29765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8062efcd52ff4b5cae783fa1441c2b8f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39357 } }
I20260619 02:38:58.757524 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:38:58.757767 29760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:38:58.758756 29766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8062efcd52ff4b5cae783fa1441c2b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8062efcd52ff4b5cae783fa1441c2b8f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39357 } } }
I20260619 02:38:58.758896 29766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:58.759227 29767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8062efcd52ff4b5cae783fa1441c2b8f. Latest consensus state: current_term: 1 leader_uuid: "8062efcd52ff4b5cae783fa1441c2b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8062efcd52ff4b5cae783fa1441c2b8f" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 39357 } } }
I20260619 02:38:58.759466 29767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f [sys.catalog]: This master's current role is: LEADER
I20260619 02:38:58.760324 29778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:38:58.763736 29778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:38:58.764899 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:38:58.760073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35203 (local address 127.7.194.254:39357)
0619 02:38:58.760155 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 02:38:58.760159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:38:58.760629 (+ 470us) server_negotiation.cc:407] Connection header received
0619 02:38:58.760783 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:38:58.760786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:38:58.760831 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:38:58.760885 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:38:58.761910 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:58.762424 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:58.763248 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:38:58.763484 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:38:58.763593 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:38:58.764429 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:38:58.764433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:38:58.764434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:38:58.764445 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:38:58.764465 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:38:58.764475 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:38:58.764584 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:38:58.764721 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 02:38:58.764754 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:38:58.769291 29778 catalog_manager.cc:1347] Generated new cluster ID: a4e51a019f4e456b833d2444a5ff4cb7
I20260619 02:38:58.769356 29778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:38:58.807432 29778 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:38:58.808454 29778 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:38:58.819075 29778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8062efcd52ff4b5cae783fa1441c2b8f: Generated new TSK 0
I20260619 02:38:58.819973 29778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:38:58.835868 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 29692
W20260619 02:38:58.840638 29677 connection.cc:535] client connection to 127.7.194.254:39357 recv error: Network error: failed to read from TLS socket (remote: 127.7.194.254:39357): Connection reset by peer (error 104)
2026-06-19T02:38:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:38:58 dist-test-slave-9tkz krb5kdc[29791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:38:58 dist-test-slave-9tkz krb5kdc[29791](info): set up 2 sockets
Jun 19 02:38:58 dist-test-slave-9tkz krb5kdc[29791](info): commencing operation
krb5kdc: starting...
W20260619 02:38:59.865975 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:38:59 dist-test-slave-9tkz krb5kdc[29791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:38:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:38:59.984771 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:35423
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:35423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:39:00.093942 29804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:00.094237 29804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:00.094316 29804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:39:00.098284 29804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:39:00.098376 29804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:39:00.098425 29804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:39:00.098501 29804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:00.098542 29804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:00.103951 29804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:35423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:35423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:00.105001 29804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:00.106038 29804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:39:00.114573 29809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:00.114557 29810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:00.114559 29812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:00.114921 29804 server_base.cc:1034] running on GCE node
I20260619 02:39:00.115468 29804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:00.116648 29804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:00.117846 29804 hybrid_clock.cc:648] HybridClock initialized: now 1781836740117822 us; error 48 us; skew 500 ppm
Jun 19 02:39:00 dist-test-slave-9tkz krb5kdc[29791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:00.120617 29804 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:39:00.121498 29804 webserver.cc:460] Webserver started at http://127.7.194.254:40171/ using document root <none> and password file <none>
I20260619 02:39:00.122110 29804 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:00.122205 29804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:00.122452 29804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:00.124485 29804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "a8a7c37fb218482db6fe173f95d11b8a"
format_stamp: "Formatted at 2026-06-19 02:39:00 on dist-test-slave-9tkz"
server_key: "09b160148cca85f1a8893a23a26651c6"
server_key_iv: "d58eaace9b7121e885d6d143e500dba9"
server_key_version: "encryptionkey@0"
I20260619 02:39:00.125061 29804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "a8a7c37fb218482db6fe173f95d11b8a"
format_stamp: "Formatted at 2026-06-19 02:39:00 on dist-test-slave-9tkz"
server_key: "09b160148cca85f1a8893a23a26651c6"
server_key_iv: "d58eaace9b7121e885d6d143e500dba9"
server_key_version: "encryptionkey@0"
I20260619 02:39:00.128780 29804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260619 02:39:00.131237 29819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:00.132431 29804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:39:00.132577 29804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "a8a7c37fb218482db6fe173f95d11b8a"
format_stamp: "Formatted at 2026-06-19 02:39:00 on dist-test-slave-9tkz"
server_key: "09b160148cca85f1a8893a23a26651c6"
server_key_iv: "d58eaace9b7121e885d6d143e500dba9"
server_key_version: "encryptionkey@0"
I20260619 02:39:00.132691 29804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:00.137468 29804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:00.140554 29804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:00.140749 29804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:00.147696 29804 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:35423
I20260619 02:39:00.147723 29871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:35423 every 8 connection(s)
I20260619 02:39:00.148617 29804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:39:00.150753 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 29804
I20260619 02:39:00.150895 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:39:00.151162 7947 external_mini_cluster.cc:1428] Setting key 239b4a3ea6e0afdb82a31009884c7bec
I20260619 02:39:00.151494 29872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:00.156592 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:00.152557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52549 (local address 127.7.194.254:35423)
0619 02:39:00.152980 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:00.152999 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:00.153043 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:39:00.153888 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:00.153912 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:00.154289 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:00.154665 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:00.155421 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:00.155426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:00.155437 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:39:00.155448 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:00.155503 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:39:00.155521 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:00.155671 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:00.155772 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 02:39:00.155927 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":123,"threads_started":1}
I20260619 02:39:00.158571 29872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a: Bootstrap starting.
I20260619 02:39:00.159444 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:39:00.161142 29872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:00.162133 29872 log.cc:824] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:00.163101 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:00.161438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42431 (local address 127.7.194.254:35423)
0619 02:39:00.161594 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:00.161599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:00.161785 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:39:00.161907 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:00.161909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:00.161939 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:00.162004 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:00.162703 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:00.162715 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:00.162717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:39:00.162729 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:00.162754 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:39:00.162771 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:00.162862 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:00.162946 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 02:39:00.162984 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:39:00.164292 29872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a: No bootstrap required, opened a new log
I20260619 02:39:00.167881 29872 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a7c37fb218482db6fe173f95d11b8a" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35423 } }
I20260619 02:39:00.168090 29872 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:39:00.168169 29872 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a7c37fb218482db6fe173f95d11b8a, State: Initialized, Role: FOLLOWER
I20260619 02:39:00.168598 29872 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a7c37fb218482db6fe173f95d11b8a" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35423 } }
I20260619 02:39:00.168780 29872 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:39:00.168877 29872 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:39:00.168982 29872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:00.169960 29872 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a7c37fb218482db6fe173f95d11b8a" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35423 } }
I20260619 02:39:00.170245 29872 leader_election.cc:302] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8a7c37fb218482db6fe173f95d11b8a; no voters:
I20260619 02:39:00.170499 29872 leader_election.cc:288] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:39:00.170617 29882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:39:00.170833 29882 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [term 1 LEADER]: Becoming Leader. State: Replica: a8a7c37fb218482db6fe173f95d11b8a, State: Running, Role: LEADER
I20260619 02:39:00.171190 29882 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a7c37fb218482db6fe173f95d11b8a" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35423 } }
I20260619 02:39:00.171702 29872 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:00.172921 29883 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8a7c37fb218482db6fe173f95d11b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a7c37fb218482db6fe173f95d11b8a" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35423 } } }
I20260619 02:39:00.172974 29884 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a7c37fb218482db6fe173f95d11b8a. Latest consensus state: current_term: 1 leader_uuid: "a8a7c37fb218482db6fe173f95d11b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a7c37fb218482db6fe173f95d11b8a" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 35423 } } }
I20260619 02:39:00.173036 29883 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:00.173065 29884 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:00.173430 29891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:39:00.176188 29891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:39:00.181751 29891 catalog_manager.cc:1347] Generated new cluster ID: 7d327a7fd64d4025af6102c0db3e44b6
I20260619 02:39:00.181818 29891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:39:00.211346 29891 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:39:00.211722 29891 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:39:00.227592 29891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8a7c37fb218482db6fe173f95d11b8a: Generated new TSK 0
I20260619 02:39:00.228513 29891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:39:00.234200 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 29804
2026-06-19T02:39:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:39:00 dist-test-slave-9tkz krb5kdc[29903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:39:00 dist-test-slave-9tkz krb5kdc[29903](info): set up 2 sockets
Jun 19 02:39:00 dist-test-slave-9tkz krb5kdc[29903](info): commencing operation
krb5kdc: starting...
W20260619 02:39:01.265978 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:39:01 dist-test-slave-9tkz krb5kdc[29903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:39:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:39:01.384656 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:34967
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:34967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:39:01.495860 29916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:01.496168 29916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:01.496261 29916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:39:01.500445 29916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:39:01.500538 29916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:39:01.500574 29916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:39:01.500630 29916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:01.500669 29916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:01.506107 29916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:34967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:34967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:01.507229 29916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:01.508306 29916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:39:01.517004 29924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:01.516978 29921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:01.516990 29922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:01.517467 29916 server_base.cc:1034] running on GCE node
I20260619 02:39:01.518040 29916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:01.519132 29916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:01.520294 29916 hybrid_clock.cc:648] HybridClock initialized: now 1781836741520270 us; error 34 us; skew 500 ppm
Jun 19 02:39:01 dist-test-slave-9tkz krb5kdc[29903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836741, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:01.528179 29916 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:39:01.529126 29916 webserver.cc:460] Webserver started at http://127.7.194.254:34449/ using document root <none> and password file <none>
I20260619 02:39:01.529760 29916 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:01.529830 29916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:01.530082 29916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:01.532186 29916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "358c4923c68e4eae9844b2fa20641e39"
format_stamp: "Formatted at 2026-06-19 02:39:01 on dist-test-slave-9tkz"
server_key: "937fa46a47ea4d7dff8377f361203c04"
server_key_iv: "d4242e9428b00854cfe47ef9d952aa3c"
server_key_version: "encryptionkey@0"
I20260619 02:39:01.532729 29916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "358c4923c68e4eae9844b2fa20641e39"
format_stamp: "Formatted at 2026-06-19 02:39:01 on dist-test-slave-9tkz"
server_key: "937fa46a47ea4d7dff8377f361203c04"
server_key_iv: "d4242e9428b00854cfe47ef9d952aa3c"
server_key_version: "encryptionkey@0"
I20260619 02:39:01.537065 29916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 02:39:01.539739 29931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:01.541087 29916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:39:01.541225 29916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "358c4923c68e4eae9844b2fa20641e39"
format_stamp: "Formatted at 2026-06-19 02:39:01 on dist-test-slave-9tkz"
server_key: "937fa46a47ea4d7dff8377f361203c04"
server_key_iv: "d4242e9428b00854cfe47ef9d952aa3c"
server_key_version: "encryptionkey@0"
I20260619 02:39:01.541338 29916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:01.559144 29916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:01.562873 29916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:01.563084 29916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:01.570122 29916 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:34967
I20260619 02:39:01.570187 29983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:34967 every 8 connection(s)
I20260619 02:39:01.571184 29916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:39:01.574016 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:01.579516 29984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39: Bootstrap starting.
I20260619 02:39:01.581036 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 29916
I20260619 02:39:01.581172 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:39:01.581470 7947 external_mini_cluster.cc:1428] Setting key b9558e406dc06757d5a95dd94b0a162e
I20260619 02:39:01.582324 29984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:01.583094 29984 log.cc:824] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:01.585253 29984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39: No bootstrap required, opened a new log
I20260619 02:39:01.589605 29984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358c4923c68e4eae9844b2fa20641e39" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34967 } }
I20260619 02:39:01.589967 29984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:39:01.590025 29984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 358c4923c68e4eae9844b2fa20641e39, State: Initialized, Role: FOLLOWER
I20260619 02:39:01.590507 29984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358c4923c68e4eae9844b2fa20641e39" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34967 } }
I20260619 02:39:01.590677 29984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:39:01.590761 29984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 19 02:39:01 dist-test-slave-9tkz krb5kdc[29903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:39:01.590894 29984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:01.591825 29984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358c4923c68e4eae9844b2fa20641e39" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34967 } }
I20260619 02:39:01.592180 29984 leader_election.cc:302] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 358c4923c68e4eae9844b2fa20641e39; no voters:
I20260619 02:39:01.592556 29984 leader_election.cc:288] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:39:01.592697 29989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:39:01.592950 29989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [term 1 LEADER]: Becoming Leader. State: Replica: 358c4923c68e4eae9844b2fa20641e39, State: Running, Role: LEADER
I20260619 02:39:01.593324 29989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358c4923c68e4eae9844b2fa20641e39" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34967 } }
I20260619 02:39:01.594028 29984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:01.594743 29991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 358c4923c68e4eae9844b2fa20641e39. Latest consensus state: current_term: 1 leader_uuid: "358c4923c68e4eae9844b2fa20641e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358c4923c68e4eae9844b2fa20641e39" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34967 } } }
I20260619 02:39:01.594940 29991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:01.595207 29990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "358c4923c68e4eae9844b2fa20641e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358c4923c68e4eae9844b2fa20641e39" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 34967 } } }
I20260619 02:39:01.595322 29990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39 [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:01.597223 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:01.582655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48351 (local address 127.7.194.254:34967)
0619 02:39:01.583152 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:01.583188 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:01.583244 (+ 56us) server_negotiation.cc:407] Connection header received
0619 02:39:01.584038 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:01.584058 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:01.584396 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:01.584724 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:01.585694 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:01.586699 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:01.587941 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:01.588413 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:01.588528 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:01.591503 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:01.591510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:01.591524 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:01.591552 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:01.593743 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:01.594305 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:01.594311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:01.594316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:01.594469 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:01.594821 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:01.594826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:01.594830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:01.595424 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:01.595639 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:01.595913 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 02:39:01.596228 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":168,"threads_started":1}
W20260619 02:39:01.598937 30004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:39:01.599026 30004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:39:01.599123 29998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:39:01.599993 29998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:39:01.600657 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:39:01.605459 29998 catalog_manager.cc:1347] Generated new cluster ID: 2a5a1e0b4caa41c4937a8b5d5f17ab57
I20260619 02:39:01.605517 29998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:39:01.611773 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:01.602892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57175 (local address 127.7.194.254:34967)
0619 02:39:01.602999 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:01.603003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:01.603236 (+ 233us) server_negotiation.cc:407] Connection header received
0619 02:39:01.603352 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:01.603356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:01.603402 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:01.603478 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:01.604254 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:01.605123 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:01.606355 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:01.606707 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:01.606797 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:01.608450 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:01.608454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:01.608457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:01.608480 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:01.610123 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:01.610599 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:01.610602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:01.610604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:01.610681 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:01.610964 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:01.610967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:01.610969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:01.611152 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:01.611267 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:01.611512 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 02:39:01.611618 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:39:01.633345 29998 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:39:01.634387 29998 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:39:01.642016 29998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 358c4923c68e4eae9844b2fa20641e39: Generated new TSK 0
I20260619 02:39:01.642930 29998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:39:01.653347 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 29916
2026-06-19T02:39:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:39:01 dist-test-slave-9tkz krb5kdc[30015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:39:01 dist-test-slave-9tkz krb5kdc[30015](info): set up 2 sockets
Jun 19 02:39:01 dist-test-slave-9tkz krb5kdc[30015](info): commencing operation
krb5kdc: starting...
W20260619 02:39:02.689976 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:39:02 dist-test-slave-9tkz krb5kdc[30015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:39:02.808333 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:41795
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:45361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:39:02.917984 30028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:02.918323 30028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:02.918411 30028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:39:02.922474 30028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:39:02.922575 30028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:39:02.922616 30028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:39:02.922676 30028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:02.922714 30028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:02.928123 30028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:45361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:02.929275 30028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:02.930378 30028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:39:02.939018 30034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:02.939126 30028 server_base.cc:1034] running on GCE node
W20260619 02:39:02.939018 30033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:02.939039 30036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:02.939766 30028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:02.940892 30028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:02.942135 30028 hybrid_clock.cc:648] HybridClock initialized: now 1781836742942122 us; error 46 us; skew 500 ppm
Jun 19 02:39:02 dist-test-slave-9tkz krb5kdc[30015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:02.944860 30028 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:39:02.945813 30028 webserver.cc:460] Webserver started at http://127.7.194.254:46109/ using document root <none> and password file <none>
I20260619 02:39:02.946384 30028 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:02.946473 30028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:02.946702 30028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:02.948798 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "1f076b2b50d64ea3b944e80d02e8b78b"
format_stamp: "Formatted at 2026-06-19 02:39:02 on dist-test-slave-9tkz"
server_key: "7317b926622fe51b2ff428012b97d8d1"
server_key_iv: "7f5a70d7d13250ee1cc8b7afdb8fe837"
server_key_version: "encryptionkey@0"
I20260619 02:39:02.949394 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "1f076b2b50d64ea3b944e80d02e8b78b"
format_stamp: "Formatted at 2026-06-19 02:39:02 on dist-test-slave-9tkz"
server_key: "7317b926622fe51b2ff428012b97d8d1"
server_key_iv: "7f5a70d7d13250ee1cc8b7afdb8fe837"
server_key_version: "encryptionkey@0"
I20260619 02:39:02.953112 30028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:39:02.955621 30043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:02.956813 30028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:39:02.956964 30028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "1f076b2b50d64ea3b944e80d02e8b78b"
format_stamp: "Formatted at 2026-06-19 02:39:02 on dist-test-slave-9tkz"
server_key: "7317b926622fe51b2ff428012b97d8d1"
server_key_iv: "7f5a70d7d13250ee1cc8b7afdb8fe837"
server_key_version: "encryptionkey@0"
I20260619 02:39:02.957077 30028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:02.975171 30028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:02.978587 30028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:02.978801 30028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:02.985333 30028 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:41795
I20260619 02:39:02.985442 30095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:41795 every 8 connection(s)
I20260619 02:39:02.986388 30028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:39:02.989194 30096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:02.994728 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 30028
I20260619 02:39:02.994881 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:39:02.995002 30096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b: Bootstrap starting.
I20260619 02:39:02.995182 7947 external_mini_cluster.cc:1428] Setting key 593d930c4805cf3105de022b01bdf2fb
I20260619 02:39:02.997714 30096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:02.998468 30096 log.cc:824] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:03.000384 30096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b: No bootstrap required, opened a new log
I20260619 02:39:03.004267 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:02.996606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47121 (local address 127.7.194.254:41795)
0619 02:39:02.997101 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:02.997116 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:02.997150 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:39:02.997933 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:02.997953 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:02.998314 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:02.998669 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:02.999077 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:03.000042 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:03.001007 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:03.001490 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:03.001706 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:03.002533 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:03.002538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:03.002551 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:39:03.002937 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:03.002991 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:39:03.003013 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:03.003089 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:03.003291 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 02:39:03.003512 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":121,"threads_started":1}
I20260619 02:39:03.004506 30096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f076b2b50d64ea3b944e80d02e8b78b" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41795 } }
I20260619 02:39:03.004777 30096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:39:03.004843 30096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f076b2b50d64ea3b944e80d02e8b78b, State: Initialized, Role: FOLLOWER
I20260619 02:39:03.005314 30096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f076b2b50d64ea3b944e80d02e8b78b" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41795 } }
I20260619 02:39:03.005491 30096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:39:03.005599 30096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:39:03.005721 30096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:03.006867 30096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f076b2b50d64ea3b944e80d02e8b78b" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41795 } }
I20260619 02:39:03.007261 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:39:03.007251 30096 leader_election.cc:302] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f076b2b50d64ea3b944e80d02e8b78b; no voters:
I20260619 02:39:03.007575 30096 leader_election.cc:288] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:39:03.007682 30101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:39:03.007930 30101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [term 1 LEADER]: Becoming Leader. State: Replica: 1f076b2b50d64ea3b944e80d02e8b78b, State: Running, Role: LEADER
I20260619 02:39:03.008312 30101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f076b2b50d64ea3b944e80d02e8b78b" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41795 } }
I20260619 02:39:03.008910 30096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:03.009739 30101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f076b2b50d64ea3b944e80d02e8b78b. Latest consensus state: current_term: 1 leader_uuid: "1f076b2b50d64ea3b944e80d02e8b78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f076b2b50d64ea3b944e80d02e8b78b" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41795 } } }
I20260619 02:39:03.009943 30101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:03.010190 30102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f076b2b50d64ea3b944e80d02e8b78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f076b2b50d64ea3b944e80d02e8b78b" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 41795 } } }
I20260619 02:39:03.010298 30102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:03.010378 30115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:39:03.013626 30115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:39:03.014672 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:03.009479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49421 (local address 127.7.194.254:41795)
0619 02:39:03.009615 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:03.009620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:03.009999 (+ 379us) server_negotiation.cc:407] Connection header received
0619 02:39:03.010041 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:03.010044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:03.010093 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:03.010165 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:03.010560 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:03.011435 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:03.012593 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:03.012942 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:03.013017 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:03.014188 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:03.014192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:03.014195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:39:03.014208 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:03.014235 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:39:03.014249 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:03.014295 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:03.014471 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 02:39:03.014528 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:39:03.019970 30115 catalog_manager.cc:1347] Generated new cluster ID: faaec850b800402ebf2c94671fc1d483
I20260619 02:39:03.020040 30115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:39:03.038357 30115 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:39:03.039402 30115 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:39:03.045073 30115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f076b2b50d64ea3b944e80d02e8b78b: Generated new TSK 0
I20260619 02:39:03.045897 30115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:39:03.050505 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 30028
2026-06-19T02:39:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:39:03 dist-test-slave-9tkz krb5kdc[30127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:39:03 dist-test-slave-9tkz krb5kdc[30127](info): set up 2 sockets
Jun 19 02:39:03 dist-test-slave-9tkz krb5kdc[30127](info): commencing operation
krb5kdc: starting...
W20260619 02:39:04.082015 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:39:04 dist-test-slave-9tkz krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:39:04.206106 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:46689
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:46689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:39:04.320715 30140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:04.321048 30140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:04.321139 30140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:39:04.325289 30140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:39:04.325403 30140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:39:04.325443 30140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:39:04.325500 30140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:04.325536 30140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:04.331036 30140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:46689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:46689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:04.332173 30140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:04.333235 30140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:39:04.342048 30148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:04.342094 30146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:04.342094 30145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:04.342242 30140 server_base.cc:1034] running on GCE node
I20260619 02:39:04.342923 30140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:04.344058 30140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:04.345242 30140 hybrid_clock.cc:648] HybridClock initialized: now 1781836744345237 us; error 46 us; skew 500 ppm
Jun 19 02:39:04 dist-test-slave-9tkz krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836744, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:04.347956 30140 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:39:04.348894 30140 webserver.cc:460] Webserver started at http://127.7.194.254:40213/ using document root <none> and password file <none>
I20260619 02:39:04.349524 30140 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:04.349660 30140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:04.349946 30140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:04.352305 30140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "97bdba499b74474690c580e24c9e75c9"
format_stamp: "Formatted at 2026-06-19 02:39:04 on dist-test-slave-9tkz"
server_key: "31d37ba0a5b942055bf6d54d166c88d8"
server_key_iv: "92854675e1f92a1d53dba0ef2ef5e25f"
server_key_version: "encryptionkey@0"
I20260619 02:39:04.352828 30140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "97bdba499b74474690c580e24c9e75c9"
format_stamp: "Formatted at 2026-06-19 02:39:04 on dist-test-slave-9tkz"
server_key: "31d37ba0a5b942055bf6d54d166c88d8"
server_key_iv: "92854675e1f92a1d53dba0ef2ef5e25f"
server_key_version: "encryptionkey@0"
I20260619 02:39:04.356520 30140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:39:04.358906 30155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:04.360124 30140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:39:04.360224 30140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "97bdba499b74474690c580e24c9e75c9"
format_stamp: "Formatted at 2026-06-19 02:39:04 on dist-test-slave-9tkz"
server_key: "31d37ba0a5b942055bf6d54d166c88d8"
server_key_iv: "92854675e1f92a1d53dba0ef2ef5e25f"
server_key_version: "encryptionkey@0"
I20260619 02:39:04.360311 30140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:04.364818 30140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:04.367638 30140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:04.367782 30140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:04.375583 30140 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:46689
I20260619 02:39:04.375597 30207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:46689 every 8 connection(s)
I20260619 02:39:04.376529 30140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:39:04.379313 30208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:04.382279 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 30140
I20260619 02:39:04.382421 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:39:04.382720 7947 external_mini_cluster.cc:1428] Setting key 1bf9518a8f93682f71dcff673c46a2f2
I20260619 02:39:04.385421 30208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9: Bootstrap starting.
I20260619 02:39:04.387966 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:04.384096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46071 (local address 127.7.194.254:46689)
0619 02:39:04.384498 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:04.384514 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:04.384548 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:39:04.385216 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:04.385233 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:04.385549 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:04.385865 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:04.386718 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:04.386722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:04.386733 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:39:04.386757 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:04.386809 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:39:04.386828 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:04.386934 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:04.387064 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 02:39:04.387258 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":118,"threads_started":1}
I20260619 02:39:04.388315 30208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:04.389050 30208 log.cc:824] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:04.390755 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:39:04.391161 30208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9: No bootstrap required, opened a new log
I20260619 02:39:04.394500 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:04.392861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33819 (local address 127.7.194.254:46689)
0619 02:39:04.392980 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:04.392985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:04.393132 (+ 147us) server_negotiation.cc:407] Connection header received
0619 02:39:04.393256 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:04.393259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:04.393287 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:04.393358 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:04.394085 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:04.394088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:04.394090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:39:04.394100 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:04.394122 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:39:04.394132 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:04.394200 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:04.394330 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 02:39:04.394362 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:39:04.395530 30208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97bdba499b74474690c580e24c9e75c9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46689 } }
I20260619 02:39:04.395797 30208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:39:04.395855 30208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97bdba499b74474690c580e24c9e75c9, State: Initialized, Role: FOLLOWER
I20260619 02:39:04.396261 30208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97bdba499b74474690c580e24c9e75c9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46689 } }
I20260619 02:39:04.396406 30208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:39:04.396494 30208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:39:04.396641 30208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:04.397642 30208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97bdba499b74474690c580e24c9e75c9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46689 } }
I20260619 02:39:04.397986 30208 leader_election.cc:302] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97bdba499b74474690c580e24c9e75c9; no voters:
I20260619 02:39:04.398315 30208 leader_election.cc:288] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:39:04.398387 30218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:39:04.398622 30218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [term 1 LEADER]: Becoming Leader. State: Replica: 97bdba499b74474690c580e24c9e75c9, State: Running, Role: LEADER
I20260619 02:39:04.398977 30218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97bdba499b74474690c580e24c9e75c9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46689 } }
I20260619 02:39:04.399567 30208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:04.400557 30220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97bdba499b74474690c580e24c9e75c9. Latest consensus state: current_term: 1 leader_uuid: "97bdba499b74474690c580e24c9e75c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97bdba499b74474690c580e24c9e75c9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46689 } } }
I20260619 02:39:04.400712 30220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:04.401062 30227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:39:04.401702 30219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97bdba499b74474690c580e24c9e75c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97bdba499b74474690c580e24c9e75c9" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 46689 } } }
I20260619 02:39:04.401796 30219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:04.403961 30227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:39:04.409641 30227 catalog_manager.cc:1347] Generated new cluster ID: 4ad14f81341249159a582e966f05c6eb
I20260619 02:39:04.409713 30227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:39:04.422966 30227 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:39:04.423300 30227 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:39:04.429257 30227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97bdba499b74474690c580e24c9e75c9: Generated new TSK 0
I20260619 02:39:04.430063 30227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:39:04.439786 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 30140
2026-06-19T02:39:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1389 ms)
[----------] 18 tests from AuthTokenIssuingTest (25203 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:39:04 dist-test-slave-9tkz krb5kdc[30239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:39:04 dist-test-slave-9tkz krb5kdc[30239](info): set up 2 sockets
Jun 19 02:39:04 dist-test-slave-9tkz krb5kdc[30239](info): commencing operation
krb5kdc: starting...
W20260619 02:39:05.482764 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:39:05 dist-test-slave-9tkz krb5kdc[30239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:39:05.605026 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:33861
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:33861,127.7.194.253:33261,127.7.194.252:40225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:39:05.716043 30252 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:39:05.716432 30252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:05.716508 30252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:05.716605 30252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:39:05.720742 30252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:39:05.720854 30252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:39:05.720891 30252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:39:05.720929 30252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:05.720964 30252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:05.726455 30252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:33861,127.7.194.253:33261,127.7.194.252:40225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:05.727686 30252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:05.728813 30252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:39:05.737627 30258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:05.737649 30257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:05.737649 30260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:05.737926 30252 server_base.cc:1034] running on GCE node
I20260619 02:39:05.738602 30252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:05.739766 30252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:05.740931 30252 hybrid_clock.cc:648] HybridClock initialized: now 1781836745740918 us; error 26 us; skew 500 ppm
Jun 19 02:39:05 dist-test-slave-9tkz krb5kdc[30239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:05.743686 30252 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:39:05.744596 30252 webserver.cc:460] Webserver started at http://127.7.194.254:45383/ using document root <none> and password file <none>
I20260619 02:39:05.745177 30252 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:05.745282 30252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:05.745517 30252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:05.747752 30252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "f06a1a00e48e4880a28de70f0149581d"
format_stamp: "Formatted at 2026-06-19 02:39:05 on dist-test-slave-9tkz"
server_key: "3a116a4e9d37e903f0f8e630d1f08940"
server_key_iv: "6e8b7bd816e54dcd80e6ff6f36ed5795"
server_key_version: "encryptionkey@0"
I20260619 02:39:05.748368 30252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "f06a1a00e48e4880a28de70f0149581d"
format_stamp: "Formatted at 2026-06-19 02:39:05 on dist-test-slave-9tkz"
server_key: "3a116a4e9d37e903f0f8e630d1f08940"
server_key_iv: "6e8b7bd816e54dcd80e6ff6f36ed5795"
server_key_version: "encryptionkey@0"
I20260619 02:39:05.752277 30252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 02:39:05.754915 30267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:05.756183 30252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:39:05.756343 30252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "f06a1a00e48e4880a28de70f0149581d"
format_stamp: "Formatted at 2026-06-19 02:39:05 on dist-test-slave-9tkz"
server_key: "3a116a4e9d37e903f0f8e630d1f08940"
server_key_iv: "6e8b7bd816e54dcd80e6ff6f36ed5795"
server_key_version: "encryptionkey@0"
I20260619 02:39:05.756471 30252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:05.767628 30252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:05.770643 30252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:05.770859 30252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:05.778077 30252 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:33861
I20260619 02:39:05.778405 30319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:33861 every 8 connection(s)
I20260619 02:39:05.779110 30252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:39:05.781288 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 30252
I20260619 02:39:05.781442 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:39:05.781790 7947 external_mini_cluster.cc:1428] Setting key 103b4064b71dc329dad2cc1afbdaa36a
I20260619 02:39:05.782446 30320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:05.786628 30320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.194.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.253@KRBTEST.COM" created.
Jun 19 02:39:05 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:39:05.802568 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.788941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:33861 (local address 127.0.0.1:42366)
0619 02:39:05.789435 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:05.789488 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:05.790429 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:05.791102 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:05.791106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:05.791578 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:05.791939 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:05.791956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:05.793233 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:05.793240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:05.794266 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:05.794276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:05.794963 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:05.794967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:05.795076 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:05.795686 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:05.795704 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:05.797258 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:05.799289 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:05.799298 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:05.799309 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:05.799539 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:05.799870 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:05.799873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:05.799874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:05.799986 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:05.800687 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:05.800693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:05.800874 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 02:39:05.801056 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 02:39:05.801503 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":115,"thread_start_us":131,"threads_started":1}
I20260619 02:39:05.802568 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.788942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.7.194.254:33861)
0619 02:39:05.789512 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:05.789524 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:05.789642 (+ 118us) server_negotiation.cc:407] Connection header received
0619 02:39:05.790719 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:05.790724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:05.790914 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:05.791076 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:05.792103 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:05.793092 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:05.794410 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:05.794784 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:05.794973 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:05.797416 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:05.797422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:05.797433 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:05.797455 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:05.799159 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:05.799648 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:05.799654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:05.799655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:05.799763 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:05.800116 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:05.800121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:05.800123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:05.800552 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:05.800703 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:05.801117 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 02:39:05.801425 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":154,"threads_started":1}
I20260619 02:39:05.805287 30320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:05.806133 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.805850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33261 (local address 127.0.0.1:48062)
0619 02:39:05.805969 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:05.806043 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:39:05.806334 30270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
I20260619 02:39:05.807199 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.806976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33261 (local address 127.0.0.1:48072)
0619 02:39:05.807086 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:05.807140 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 02:39:05.807380 30320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.253:33261: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab.
Entry for principal kudu/127.7.194.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab.
I20260619 02:39:05.808594 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.253:33261
--webserver_interface=127.7.194.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab
--principal=kudu/127.7.194.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:33861,127.7.194.253:33261,127.7.194.252:40225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:39:05.856631 30320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } attempt: 1
I20260619 02:39:05.857576 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.857267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33261 (local address 127.0.0.1:48084)
0619 02:39:05.857419 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:05.857483 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 02:39:05.858295 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.858070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33261 (local address 127.0.0.1:48086)
0619 02:39:05.858186 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:05.858236 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:39:05.858425 30320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.253:33261: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
W20260619 02:39:05.922451 30327 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:39:05.922829 30327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:05.922907 30327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:05.922986 30327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:39:05.926591 30320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } attempt: 2
W20260619 02:39:05.927312 30327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:39:05.927402 30327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:39:05.927452 30327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260619 02:39:05.927443 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.927112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33261 (local address 127.0.0.1:48090)
0619 02:39:05.927264 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:05.927335 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 02:39:05.927493 30327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:05.927594 30327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:05.928263 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.928043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33261 (local address 127.0.0.1:48100)
0619 02:39:05.928150 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:05.928201 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 02:39:05.928399 30320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.253:33261: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33261: connect: Connection refused (error 111)
I20260619 02:39:05.933373 30327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:33861,127.7.194.253:33261,127.7.194.252:40225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab
--principal=kudu/127.7.194.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.253:33261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.194.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:05.934885 30327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:05.936148 30327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:39:05.944832 30333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:05.944932 30332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:05.944852 30335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:05.945042 30327 server_base.cc:1034] running on GCE node
I20260619 02:39:05.945827 30327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:05.946986 30327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:05.948176 30327 hybrid_clock.cc:648] HybridClock initialized: now 1781836745948164 us; error 36 us; skew 500 ppm
Jun 19 02:39:05 dist-test-slave-9tkz krb5kdc[30239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:05.951134 30327 init.cc:377] Logged in from keytab as kudu/127.7.194.253@KRBTEST.COM (short username kudu)
I20260619 02:39:05.952070 30327 webserver.cc:460] Webserver started at http://127.7.194.253:33783/ using document root <none> and password file <none>
I20260619 02:39:05.952665 30327 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:05.952730 30327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:05.952994 30327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:05.955279 30327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/data/instance:
uuid: "3914fd5df1a145c793e86ab7777240f5"
format_stamp: "Formatted at 2026-06-19 02:39:05 on dist-test-slave-9tkz"
server_key: "23fd837718afd6d5806d6b8d746ba12f"
server_key_iv: "492c658654fdac03df694d3911f49f65"
server_key_version: "encryptionkey@0"
I20260619 02:39:05.955868 30327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/wal/instance:
uuid: "3914fd5df1a145c793e86ab7777240f5"
format_stamp: "Formatted at 2026-06-19 02:39:05 on dist-test-slave-9tkz"
server_key: "23fd837718afd6d5806d6b8d746ba12f"
server_key_iv: "492c658654fdac03df694d3911f49f65"
server_key_version: "encryptionkey@0"
I20260619 02:39:05.959816 30327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:39:05.962431 30342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:05.963677 30327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:39:05.963832 30327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/wal
uuid: "3914fd5df1a145c793e86ab7777240f5"
format_stamp: "Formatted at 2026-06-19 02:39:05 on dist-test-slave-9tkz"
server_key: "23fd837718afd6d5806d6b8d746ba12f"
server_key_iv: "492c658654fdac03df694d3911f49f65"
server_key_version: "encryptionkey@0"
I20260619 02:39:05.963955 30327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:05.975852 30327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:05.979287 30327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:05.979507 30327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:05.986539 30327 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.253:33261
I20260619 02:39:05.986821 30394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.253:33261 every 8 connection(s)
I20260619 02:39:05.987561 30327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
I20260619 02:39:05.990418 30395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:05.994292 30395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:05.995170 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 30327
I20260619 02:39:05.995277 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-1/wal/instance
I20260619 02:39:05.995570 7947 external_mini_cluster.cc:1428] Setting key 09d7a95d3285fcffaa4741a75e418b05
WARNING: no policy specified for kudu/127.7.194.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.252@KRBTEST.COM" created.
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:39:06.012627 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.996732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.7.194.254:33861)
0619 02:39:05.996879 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:05.996883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:05.998080 (+ 1197us) server_negotiation.cc:407] Connection header received
0619 02:39:05.999209 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:05.999216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:05.999287 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:05.999399 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.000722 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.001739 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.002927 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.003361 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.003503 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.007813 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.007817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.007819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.007847 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.010275 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.010996 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.011000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.011001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.011082 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.011475 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.011479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.011481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.011678 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.011795 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.012325 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.012460 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:39:06.013621 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:05.997233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:33861 (local address 127.0.0.1:42382)
0619 02:39:05.997875 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:05.997940 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:05.998975 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:05.999564 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:05.999571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:06.000026 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:06.000547 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.000562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.001880 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.001883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.002776 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.002803 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.003566 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.003570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.003750 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.004372 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:06.004389 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:06.007598 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:06.010432 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.010442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.010465 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.010880 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.011199 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.011202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.011205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.011363 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.011785 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:06.011792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:06.012073 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 02:39:06.012277 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 02:39:06.012692 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":121,"threads_started":1}
I20260619 02:39:06.015313 30395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:06.019557 30320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } attempt: 3
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab.
Entry for principal kudu/127.7.194.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab.
I20260619 02:39:06.024410 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.252:40225
--webserver_interface=127.7.194.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab
--principal=kudu/127.7.194.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:33861,127.7.194.253:33261,127.7.194.252:40225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
I20260619 02:39:06.025666 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.015804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33261 (local address 127.0.0.1:48106)
0619 02:39:06.015941 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.015957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:06.016035 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:06.016569 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:06.016571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:06.016799 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:06.016898 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.016904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.017909 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.017912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.018526 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.018535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.019085 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.019088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.019161 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.019733 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:06.019754 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:06.021488 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:06.023662 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.023665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.023667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.023906 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.024220 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.024225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.024227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.024357 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.024959 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:06.024965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:06.025080 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 02:39:06.025198 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 02:39:06.025358 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:39:06.025799 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.015914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.7.194.253:33261)
0619 02:39:06.016240 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.016250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.016268 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:39:06.016302 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.016306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.016457 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.016563 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.016998 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.017808 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.018630 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.018990 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.019067 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.021650 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.021658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.021669 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.021689 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.023556 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.024026 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.024031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.024033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.024114 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.024465 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.024468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.024470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.024810 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.024966 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.025320 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.025468 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":150,"threads_started":1}
I20260619 02:39:06.028201 30395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:06.028999 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.028688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37192)
0619 02:39:06.028833 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.028919 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 02:39:06.029464 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
I20260619 02:39:06.030145 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.020125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.7.194.253:33261)
0619 02:39:06.020502 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.020515 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.020532 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:39:06.020569 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.020572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.020611 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.020704 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.021169 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.022006 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.023028 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.023356 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.023507 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.026282 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.026290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.026297 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.026323 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.028377 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.028858 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.028862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.028863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.028942 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.029282 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.029285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.029286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.029456 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.029591 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.029865 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.029983 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":126,"threads_started":1}
I20260619 02:39:06.030155 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.020043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33261 (local address 127.0.0.1:48118)
0619 02:39:06.020164 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.020180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:06.020271 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:06.020733 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:06.020735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:06.020928 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:06.021017 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.021023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.022109 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.022114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.022893 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.022902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.023535 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.023539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.023666 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.024107 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:06.024121 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:06.025600 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:06.028486 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.028489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.028491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.028714 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.029053 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.029060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.029065 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.029176 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.029601 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:06.029606 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:06.029704 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 02:39:06.029816 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:39:06.029980 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 02:39:06.030892 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.030689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37198)
0619 02:39:06.030796 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.030839 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 02:39:06.031076 30395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:40225: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
I20260619 02:39:06.031332 30320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:06.031997 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.031771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37208)
0619 02:39:06.031881 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.031928 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:39:06.032627 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.032398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37212)
0619 02:39:06.032529 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.032572 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 02:39:06.032738 30320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:40225: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
I20260619 02:39:06.063884 30320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } attempt: 1
I20260619 02:39:06.064754 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.064461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37228)
0619 02:39:06.064609 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.064673 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 02:39:06.065476 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.065266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37238)
0619 02:39:06.065375 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.065424 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:39:06.065620 30320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:40225: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
I20260619 02:39:06.080340 30395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } attempt: 1
I20260619 02:39:06.081143 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.080861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37246)
0619 02:39:06.080998 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.081067 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:39:06.081882 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.081671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37250)
0619 02:39:06.081775 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.081832 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:39:06.082012 30395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:40225: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
W20260619 02:39:06.136934 30402 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:39:06.137269 30402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:06.137341 30402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:06.137426 30402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:39:06.140799 30320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } attempt: 2
W20260619 02:39:06.141674 30402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260619 02:39:06.141685 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.141337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37254)
0619 02:39:06.141498 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.141593 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 02:39:06.141736 30402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:39:06.141765 30402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:39:06.141790 30402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:06.141827 30402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:06.142416 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.142186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37258)
0619 02:39:06.142296 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.142353 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:39:06.142549 30320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:40225: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
I20260619 02:39:06.147240 30402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:33861,127.7.194.253:33261,127.7.194.252:40225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab
--principal=kudu/127.7.194.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.252:40225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.194.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:06.148278 30402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:06.149339 30402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:39:06.150156 30395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } attempt: 2
I20260619 02:39:06.150979 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.150699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37270)
0619 02:39:06.150848 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.150912 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 02:39:06.151681 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.151476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37274)
0619 02:39:06.151580 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.151624 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 02:39:06.151845 30395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:40225: Network error: Client connection negotiation failed: client connection to 127.7.194.252:40225: connect: Connection refused (error 111)
W20260619 02:39:06.158311 30409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:06.158483 30402 server_base.cc:1034] running on GCE node
W20260619 02:39:06.158311 30408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:06.158331 30411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:06.159215 30402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:06.160483 30402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:06.161617 30402 hybrid_clock.cc:648] HybridClock initialized: now 1781836746161588 us; error 55 us; skew 500 ppm
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:06.164276 30402 init.cc:377] Logged in from keytab as kudu/127.7.194.252@KRBTEST.COM (short username kudu)
I20260619 02:39:06.165153 30402 webserver.cc:460] Webserver started at http://127.7.194.252:33933/ using document root <none> and password file <none>
I20260619 02:39:06.165799 30402 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:06.165890 30402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:06.166131 30402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:06.168206 30402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/data/instance:
uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d"
format_stamp: "Formatted at 2026-06-19 02:39:06 on dist-test-slave-9tkz"
server_key: "9fbbc90592993d9dd1c0baa258677a75"
server_key_iv: "d7d36e469a384216c83473f59c265242"
server_key_version: "encryptionkey@0"
I20260619 02:39:06.168792 30402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/wal/instance:
uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d"
format_stamp: "Formatted at 2026-06-19 02:39:06 on dist-test-slave-9tkz"
server_key: "9fbbc90592993d9dd1c0baa258677a75"
server_key_iv: "d7d36e469a384216c83473f59c265242"
server_key_version: "encryptionkey@0"
I20260619 02:39:06.172477 30402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:39:06.174886 30418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:06.176095 30402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:39:06.176239 30402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/wal
uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d"
format_stamp: "Formatted at 2026-06-19 02:39:06 on dist-test-slave-9tkz"
server_key: "9fbbc90592993d9dd1c0baa258677a75"
server_key_iv: "d7d36e469a384216c83473f59c265242"
server_key_version: "encryptionkey@0"
I20260619 02:39:06.176355 30402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:06.201260 30402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:06.204252 30402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:06.204466 30402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:06.211414 30402 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.252:40225
I20260619 02:39:06.211694 30470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.252:40225 every 8 connection(s)
I20260619 02:39:06.212482 30402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
I20260619 02:39:06.215301 30471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:06.218945 30471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:06.222093 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 30402
I20260619 02:39:06.222218 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781836464269074-7947-0/minicluster-data/master-2/wal/instance
I20260619 02:39:06.222519 7947 external_mini_cluster.cc:1428] Setting key b591e32fb8b317b7fbea9088724d505f
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
I20260619 02:39:06.233733 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.220749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.7.194.254:33861)
0619 02:39:06.220899 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.220903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.221703 (+ 800us) server_negotiation.cc:407] Connection header received
0619 02:39:06.223062 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.223065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.223130 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.223217 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.224695 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.225474 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.226756 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.226974 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.227095 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.229700 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.229704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.229707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.229736 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.231397 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.231928 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.231931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.231932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.232015 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.232395 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.232400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.232405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.232663 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.232777 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.233370 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.233509 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:39:06.234750 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.221081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:33861 (local address 127.0.0.1:42388)
0619 02:39:06.221508 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.221600 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:06.222675 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:06.223422 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:06.223432 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:06.223892 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:06.224528 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.224549 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.225643 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.225645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.226637 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.226647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.227132 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.227135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.227291 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.227920 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:06.227944 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:06.229529 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:06.231515 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.231525 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.231539 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.231831 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.232103 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.232106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.232108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.232302 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.232790 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:06.232795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:06.233106 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 02:39:06.233316 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 02:39:06.233690 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":114,"threads_started":1}
I20260619 02:39:06.234750 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.223197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.7.194.252:40225)
0619 02:39:06.223572 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.223580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.223601 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:39:06.223709 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.223713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.223867 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.223996 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.224892 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.226013 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.227058 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.227452 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.227531 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.230175 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.230183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.230197 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.230224 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.232133 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.232577 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.232581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.232582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.232650 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.232934 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.232937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.232939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.233186 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.233386 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.233637 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.233757 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":133,"threads_started":1}
I20260619 02:39:06.236052 30471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
I20260619 02:39:06.241693 30320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } attempt: 3
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:39:06.243072 30395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } attempt: 3
I20260619 02:39:06.245200 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.236523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33261 (local address 127.0.0.1:48120)
0619 02:39:06.236634 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.236650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:06.236721 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:06.237016 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:06.237019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:06.237212 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:06.237346 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.237354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.238170 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.238176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.238694 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.238701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.239201 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.239204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.239312 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.239758 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:06.239772 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:06.241111 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:06.243312 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.243319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.243324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.243575 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.243912 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.243915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.243917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.244005 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.244489 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:06.244493 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:06.244586 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 02:39:06.244897 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 02:39:06.245022 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:39:06.245278 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.236659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.7.194.253:33261)
0619 02:39:06.236795 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.236801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.236816 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:39:06.236856 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.236859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.236902 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.236981 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.237456 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.238060 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.238828 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.239093 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.239204 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.241250 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.241253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.241256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.241296 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.243179 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.243696 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.243700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.243702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.243791 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.244106 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.244109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.244111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.244351 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.244420 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.244770 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.245130 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:39:06.246630 30471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:06.246560 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.237348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.7.194.254:33861)
0619 02:39:06.237457 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.237461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.237474 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:39:06.237610 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.237614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.237651 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.237744 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.238331 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.238805 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.239581 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.239907 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.240015 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.242538 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.242542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.242545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.242566 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.244304 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.244768 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.244771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.244773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.244868 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.245159 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.245162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.245164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.245365 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.245606 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.246232 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.246390 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
I20260619 02:39:06.251971 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.242116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37278)
0619 02:39:06.242383 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.242397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:06.242478 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:06.242739 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:06.242743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:06.242957 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:06.243073 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.243082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.244135 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.244138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.244971 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.244979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.245593 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.245596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.245675 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.246138 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:06.246152 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:06.247573 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:06.250393 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.250396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.250398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.250648 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.250961 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.250964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.250966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.251052 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.251426 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:06.251433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:06.251541 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 02:39:06.251656 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 02:39:06.251786 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 02:39:06.252007 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.242381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.7.194.252:40225)
0619 02:39:06.242492 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.242496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.242533 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:39:06.242572 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.242576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.242622 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.242703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.243215 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.243994 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.245100 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.245452 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.245879 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.247846 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.247849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.247851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.247873 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.250269 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.250761 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.250764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.250766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.250858 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.251155 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.251158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.251159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.251339 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.251428 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.251731 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.251836 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
Jun 19 02:39:06 dist-test-slave-9tkz krb5kdc[30239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
I20260619 02:39:06.255559 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.244017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37294)
0619 02:39:06.244188 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.244202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:06.244263 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:06.244762 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:06.244765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:06.244952 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:06.245041 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.245047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.247021 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.247026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.247848 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.247859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.249024 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.249028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.249111 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.249747 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:06.249761 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:06.251296 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:06.253847 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.253850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.253853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.254093 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.254406 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.254411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.254414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.254512 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.255018 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:06.255022 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:06.255130 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 02:39:06.255222 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:39:06.255365 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 02:39:06.256172 30320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d: Bootstrap starting.
I20260619 02:39:06.256489 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.243704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.7.194.252:40225)
0619 02:39:06.244544 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.244558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.244569 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:39:06.244598 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.244600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.244636 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.245491 (+ 855us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.245599 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.246882 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.247975 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.248899 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.251657 (+ 2758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.251757 (+ 100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.251761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.251763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.251794 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.253707 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.254200 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.254203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.254204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.254297 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.254636 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.254642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.254644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.254899 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.255854 (+ 955us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.255928 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.256292 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":150,"threads_started":1}
I20260619 02:39:06.258036 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.247097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:40225 (local address 127.0.0.1:37302)
0619 02:39:06.247728 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:06.247741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:06.247803 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:06.248136 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:06.248139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:06.248337 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:06.248425 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.248434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.249414 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.249418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.250315 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:06.250329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.251061 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.251064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:06.251141 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.252457 (+ 1316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:06.252476 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:06.254044 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:06.256568 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.256572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.256575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.256798 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.257055 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:06.257057 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:06.257059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:06.257136 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:06.257575 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:06.257581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:06.257686 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 02:39:06.257786 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 02:39:06.257897 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"spinlock_wait_cycles":51072}
I20260619 02:39:06.258518 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.247261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.7.194.252:40225)
0619 02:39:06.247686 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.247695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.247870 (+ 175us) server_negotiation.cc:407] Connection header received
0619 02:39:06.247897 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.247903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.247942 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.248010 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.248539 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.249306 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.250444 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.250845 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.250949 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.254182 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.254186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.254190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.254210 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.256458 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.256890 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.256893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.256894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.256968 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.257219 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.257221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.257223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.257419 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.257506 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.258235 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.258355 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":98,"threads_started":1}
I20260619 02:39:06.258929 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.248218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.7.194.253:33261)
0619 02:39:06.248417 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.248421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.248647 (+ 226us) server_negotiation.cc:407] Connection header received
0619 02:39:06.248692 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.248696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.248748 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.248820 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.249740 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.250504 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.251525 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.251855 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.251949 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.255355 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.255358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.255361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.255383 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.256832 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.257283 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.257287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.257288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.257365 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.257704 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.257710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.257713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.257925 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.258099 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.258676 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.258776 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260619 02:39:06.259660 30395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5: Bootstrap starting.
I20260619 02:39:06.259914 30320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:06.260826 30320 log.cc:824] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:06.261407 30471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d: Bootstrap starting.
I20260619 02:39:06.262287 30395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:06.262734 30320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d: No bootstrap required, opened a new log
I20260619 02:39:06.263101 30395 log.cc:824] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:06.263819 30471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:06.264636 30471 log.cc:824] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:06.266541 30395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5: No bootstrap required, opened a new log
I20260619 02:39:06.266634 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:39:06.266623 30320 raft_consensus.cc:348] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } }
I20260619 02:39:06.266863 30320 raft_consensus.cc:729] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06a1a00e48e4880a28de70f0149581d, State: Initialized, Role: FOLLOWER
I20260619 02:39:06.267198 30471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d: No bootstrap required, opened a new log
I20260619 02:39:06.267194 30320 consensus_queue.cc:260] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } }
I20260619 02:39:06.268141 30294 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f06a1a00e48e4880a28de70f0149581d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42398
I20260619 02:39:06.268654 30294 raft_consensus.cc:482] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:39:06.268617 30320 sys_catalog.cc:564] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:06.268772 30294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:06.270182 30294 raft_consensus.cc:504] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } }
I20260619 02:39:06.271200 30294 leader_election.cc:288] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [CANDIDATE]: Term 1 election: Requested vote from peers 3914fd5df1a145c793e86ab7777240f5 (127.7.194.253:33261), 972d9ac2c8e54cabbe7b1b0a37b3651d (127.7.194.252:40225)
I20260619 02:39:06.271034 30395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } }
I20260619 02:39:06.271634 30395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3914fd5df1a145c793e86ab7777240f5, State: Initialized, Role: FOLLOWER
I20260619 02:39:06.271974 30395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } }
I20260619 02:39:06.271981 30471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } }
W20260619 02:39:06.272368 30273 tablet.cc:2369] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:39:06.272349 30471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972d9ac2c8e54cabbe7b1b0a37b3651d, State: Initialized, Role: FOLLOWER
I20260619 02:39:06.272748 30471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } }
I20260619 02:39:06.273342 30395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:06.273370 30369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f06a1a00e48e4880a28de70f0149581d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3914fd5df1a145c793e86ab7777240f5"
I20260619 02:39:06.273607 30369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:06.274850 30445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f06a1a00e48e4880a28de70f0149581d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d"
I20260619 02:39:06.275018 30369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f06a1a00e48e4880a28de70f0149581d in term 1.
I20260619 02:39:06.275095 30445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:06.275600 30481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.273094 30493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.275849 30493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:06.275851 30481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:06.276405 30445 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f06a1a00e48e4880a28de70f0149581d in term 1.
I20260619 02:39:06.274791 30471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:06.276573 30494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.276784 30494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:06.276887 30269 leader_election.cc:302] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 972d9ac2c8e54cabbe7b1b0a37b3651d, f06a1a00e48e4880a28de70f0149581d; no voters:
I20260619 02:39:06.277143 30481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:39:06.277376 30481 raft_consensus.cc:686] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [term 1 LEADER]: Becoming Leader. State: Replica: f06a1a00e48e4880a28de70f0149581d, State: Running, Role: LEADER
I20260619 02:39:06.277837 30481 consensus_queue.cc:237] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } }
I20260619 02:39:06.273995 30252 master_runner.cc:186] Error getting master registration for 127.7.194.253:33261: OK, instance_id { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" instance_seqno: 1781836745984486 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:39:06.280578 30252 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 02:39:06.282181 30517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:39:06.282332 30517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:39:06.282719 30507 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f06a1a00e48e4880a28de70f0149581d. Latest consensus state: current_term: 1 leader_uuid: "f06a1a00e48e4880a28de70f0149581d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.282867 30507 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:06.283306 30524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:39:06.287071 30524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 02:39:06.290468 30505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:39:06.290555 30505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:39:06.299937 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.286861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.7.194.253:33261)
0619 02:39:06.286973 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.286977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.289715 (+ 2738us) server_negotiation.cc:407] Connection header received
0619 02:39:06.290860 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.290864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.290915 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.290973 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.291666 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.292444 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.293657 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.293989 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.294059 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.295938 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.295942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.295945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.295969 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.297707 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.298196 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.298209 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.298211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.298300 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.298849 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.298855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.298858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.299083 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.299193 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.299606 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.299724 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:39:06.306461 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.287457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.7.194.252:40225)
0619 02:39:06.287581 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.287585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.291866 (+ 4281us) server_negotiation.cc:407] Connection header received
0619 02:39:06.292498 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.292505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.292555 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.292604 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.295426 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.296237 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.298414 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.298807 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.298907 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.302593 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.302597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.302600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.302626 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.304458 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.304955 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.304959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.304961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.305047 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.305395 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.305399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.305402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.305652 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.305797 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.306150 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.306282 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:39:06.307337 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:06.283823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.7.194.254:33861)
0619 02:39:06.283930 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.283935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.289494 (+ 5559us) server_negotiation.cc:407] Connection header received
0619 02:39:06.289924 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.289928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.289978 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.290035 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.293495 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.294248 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.297447 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:06.297820 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:06.297910 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:06.303537 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:06.303543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:06.303546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:06.303575 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:06.305290 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.305780 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.305783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.305785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.305884 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:06.306178 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:06.306181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:06.306183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:06.306386 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:06.306492 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:06.307079 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 02:39:06.307200 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:39:06.315531 30369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [term 1 FOLLOWER]: Refusing update from remote peer f06a1a00e48e4880a28de70f0149581d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:39:06.315588 30445 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [term 1 FOLLOWER]: Refusing update from remote peer f06a1a00e48e4880a28de70f0149581d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:39:06.316488 30507 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:39:06.317900 30507 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [LEADER]: Connected to new peer: Peer: permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:39:06.321586 30493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f06a1a00e48e4880a28de70f0149581d. Latest consensus state: current_term: 1 leader_uuid: "f06a1a00e48e4880a28de70f0149581d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.321707 30493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:06.322144 30494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f06a1a00e48e4880a28de70f0149581d. Latest consensus state: current_term: 1 leader_uuid: "f06a1a00e48e4880a28de70f0149581d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.322268 30494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:06.333412 30481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f06a1a00e48e4880a28de70f0149581d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.333566 30481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:06.333832 30530 mvcc.cc:205] Tried to move back new op lower bound from 7298403312855105536 to 7298403312761257984. Current Snapshot: MvccSnapshot[applied={T|T < 7298403312855105536}]
I20260619 02:39:06.333902 30493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f06a1a00e48e4880a28de70f0149581d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.334020 30493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:06.334718 30531 mvcc.cc:205] Tried to move back new op lower bound from 7298403312855105536 to 7298403312761257984. Current Snapshot: MvccSnapshot[applied={T|T < 7298403312855105536}]
I20260619 02:39:06.337006 30507 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f06a1a00e48e4880a28de70f0149581d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.337124 30507 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:06.338153 30494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f06a1a00e48e4880a28de70f0149581d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06a1a00e48e4880a28de70f0149581d" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 33861 } } peers { permanent_uuid: "3914fd5df1a145c793e86ab7777240f5" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33261 } } peers { permanent_uuid: "972d9ac2c8e54cabbe7b1b0a37b3651d" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 40225 } } }
I20260619 02:39:06.338277 30494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:06.338789 30533 mvcc.cc:205] Tried to move back new op lower bound from 7298403312855105536 to 7298403312761257984. Current Snapshot: MvccSnapshot[applied={T|T < 7298403312855105536}]
I20260619 02:39:06.341843 30524 catalog_manager.cc:1347] Generated new cluster ID: c46328145db244159b3fe377e8800085
I20260619 02:39:06.341907 30524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:39:06.376407 30524 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:39:06.377791 30524 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:39:06.401371 30524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f06a1a00e48e4880a28de70f0149581d: Generated new TSK 0
I20260619 02:39:06.402329 30524 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:39:06.442067 30324 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.438554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.7.194.254:33861)
0619 02:39:06.438704 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.438710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.440704 (+ 1994us) server_negotiation.cc:407] Connection header received
0619 02:39:06.440996 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.441000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.441061 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.441137 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.441902 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42416: BlockingRecv error: recv got EOF from 127.0.0.1:42416 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 02:39:06.443526 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.439390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.7.194.253:33261)
0619 02:39:06.439513 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.439520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.442482 (+ 2962us) server_negotiation.cc:407] Connection header received
0619 02:39:06.442669 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.442672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.442744 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.442800 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.443406 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48158: BlockingRecv error: recv got EOF from 127.0.0.1:48158 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 02:39:06.444984 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.440099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.7.194.252:40225)
0619 02:39:06.440259 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.440264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.444036 (+ 3772us) server_negotiation.cc:407] Connection header received
0619 02:39:06.444148 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.444151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.444208 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.444299 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.444863 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37314: BlockingRecv error: recv got EOF from 127.0.0.1:37314 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 02:39:06.450240 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.447597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.7.194.253:33261)
0619 02:39:06.447714 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.447718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.449081 (+ 1363us) server_negotiation.cc:407] Connection header received
0619 02:39:06.449359 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.449363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.449457 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.449518 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.450133 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48164: BlockingRecv error: recv got EOF from 127.0.0.1:48164 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 02:39:06.455691 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.454081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48174 (local address 127.7.194.253:33261)
0619 02:39:06.454205 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.454210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.454768 (+ 558us) server_negotiation.cc:407] Connection header received
0619 02:39:06.454966 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.454970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.455019 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.455067 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.455589 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48174: BlockingRecv error: recv got EOF from 127.0.0.1:48174 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 02:39:06.461779 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.459930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.7.194.253:33261)
0619 02:39:06.460070 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.460074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.460741 (+ 667us) server_negotiation.cc:407] Connection header received
0619 02:39:06.461005 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.461009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.461062 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.461115 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.461666 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48190: BlockingRecv error: recv got EOF from 127.0.0.1:48190 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 02:39:06.470005 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.468253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.7.194.253:33261)
0619 02:39:06.468415 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.468420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.469017 (+ 597us) server_negotiation.cc:407] Connection header received
0619 02:39:06.469221 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.469224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.469275 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.469326 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.469896 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48198: BlockingRecv error: recv got EOF from 127.0.0.1:48198 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 02:39:06.482318 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.480527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.7.194.253:33261)
0619 02:39:06.480661 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.480666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.481267 (+ 601us) server_negotiation.cc:407] Connection header received
0619 02:39:06.481458 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.481461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.481511 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.481596 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.482199 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48208: BlockingRecv error: recv got EOF from 127.0.0.1:48208 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 02:39:06.502070 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.500653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.7.194.253:33261)
0619 02:39:06.500800 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.500805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.501079 (+ 274us) server_negotiation.cc:407] Connection header received
0619 02:39:06.501257 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.501260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.501315 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.501409 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.501944 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48212: BlockingRecv error: recv got EOF from 127.0.0.1:48212 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 02:39:06.537662 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.536417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.7.194.253:33261)
0619 02:39:06.536553 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.536561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.536790 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:39:06.536902 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.536905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.536956 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.537036 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.537511 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48222: BlockingRecv error: recv got EOF from 127.0.0.1:48222 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 02:39:06.604943 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.603804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.7.194.253:33261)
0619 02:39:06.603940 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.603944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.604176 (+ 232us) server_negotiation.cc:407] Connection header received
0619 02:39:06.604300 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.604302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.604354 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.604428 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.604845 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48234: BlockingRecv error: recv got EOF from 127.0.0.1:48234 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 02:39:06.736214 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.735150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.7.194.253:33261)
0619 02:39:06.735306 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.735310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.735467 (+ 157us) server_negotiation.cc:407] Connection header received
0619 02:39:06.735569 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.735571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.735620 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.735706 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.736118 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48246: BlockingRecv error: recv got EOF from 127.0.0.1:48246 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 02:39:06.995697 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:06.994521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.7.194.253:33261)
0619 02:39:06.994678 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:06.994682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:06.994831 (+ 149us) server_negotiation.cc:407] Connection header received
0619 02:39:06.994963 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:06.994965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:06.995011 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:06.995084 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:06.995581 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48250: BlockingRecv error: recv got EOF from 127.0.0.1:48250 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:39:07.283756 30517 catalog_manager.cc:1259] Loaded cluster ID: c46328145db244159b3fe377e8800085
I20260619 02:39:07.283844 30517 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d: loading cluster ID for follower catalog manager: success
I20260619 02:39:07.285476 30517 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d: acquiring CA information for follower catalog manager: success
I20260619 02:39:07.286700 30517 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 972d9ac2c8e54cabbe7b1b0a37b3651d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:39:07.291813 30505 catalog_manager.cc:1259] Loaded cluster ID: c46328145db244159b3fe377e8800085
I20260619 02:39:07.291882 30505 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5: loading cluster ID for follower catalog manager: success
I20260619 02:39:07.293569 30505 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5: acquiring CA information for follower catalog manager: success
I20260619 02:39:07.294677 30505 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3914fd5df1a145c793e86ab7777240f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:39:07.513700 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:07.509958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.7.194.253:33261)
0619 02:39:07.510214 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:07.510218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:07.510274 (+ 56us) server_negotiation.cc:407] Connection header received
0619 02:39:07.510431 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:07.510434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:07.510555 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:07.510666 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:07.510896 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.511423 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.512231 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.512460 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.512541 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:07.512815 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:07.513252 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:07.513358 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:07.513414 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 02:39:07.513460 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
I20260619 02:39:07.520313 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:07.516641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.7.194.252:40225)
0619 02:39:07.516878 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:07.516881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:07.516894 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:39:07.516971 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:07.516973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:07.517085 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:07.517182 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:07.517361 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.517933 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.518723 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.518988 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.519100 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:07.519402 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:07.519925 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:07.520015 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:07.520139 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:39:07.520192 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260619 02:39:07.527459 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:07.523479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.7.194.253:33261)
0619 02:39:07.523634 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:07.523640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:07.523919 (+ 279us) server_negotiation.cc:407] Connection header received
0619 02:39:07.524022 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:07.524025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:07.524078 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:07.524145 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:07.524421 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.525217 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.526184 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.526538 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.526656 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:07.526913 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:07.527001 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:07.527051 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:07.527233 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 02:39:07.527316 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:39:07.527622 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:07.523297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.7.194.252:40225)
0619 02:39:07.523408 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:07.523412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:07.523654 (+ 242us) server_negotiation.cc:407] Connection header received
0619 02:39:07.523882 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:07.523885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:07.523939 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:07.524010 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:07.524520 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.525030 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.526120 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.526399 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.526513 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:07.526948 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:07.527048 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:07.527124 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:07.527367 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 02:39:07.527444 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:39:07.528443 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:07.523171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.7.194.254:33861)
0619 02:39:07.523429 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:07.523434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:07.523793 (+ 359us) server_negotiation.cc:407] Connection header received
0619 02:39:07.523892 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:07.523897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:07.524058 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:07.524191 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:07.524661 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.525408 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.526512 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:07.526876 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:07.526989 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:07.527488 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:07.527879 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:07.527959 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:07.528238 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 02:39:07.528300 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":66,"threads_started":1}
I20260619 02:39:07.531631 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 30252
I20260619 02:39:07.538025 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 30327
I20260619 02:39:07.544121 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 30402
2026-06-19T02:39:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3105 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:39:07 dist-test-slave-9tkz krb5kdc[30620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:39:07 dist-test-slave-9tkz krb5kdc[30620](info): set up 2 sockets
Jun 19 02:39:07 dist-test-slave-9tkz krb5kdc[30620](info): commencing operation
krb5kdc: starting...
W20260619 02:39:08.582341 7947 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/test-user.keytab.
Jun 19 02:39:08 dist-test-slave-9tkz krb5kdc[30620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:39:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
Entry for principal kudu/127.7.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab.
I20260619 02:39:08.702725 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38217
--webserver_interface=127.7.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38217,127.7.194.253:33593,127.7.194.252:33101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:39:08.815495 30633 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:39:08.815817 30633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:08.815843 30633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:08.815932 30633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:39:08.820881 30633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:39:08.820972 30633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:39:08.820994 30633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:39:08.821012 30633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:08.821028 30633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:08.826922 30633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38217,127.7.194.253:33593,127.7.194.252:33101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.254.keytab
--principal=kudu/127.7.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.254:38217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:08.828106 30633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:08.829262 30633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:39:08.838660 30638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:08.838649 30639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:08.838660 30641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:08.839262 30633 server_base.cc:1034] running on GCE node
I20260619 02:39:08.839884 30633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:08.841037 30633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:08.842222 30633 hybrid_clock.cc:648] HybridClock initialized: now 1781836748842205 us; error 35 us; skew 500 ppm
Jun 19 02:39:08 dist-test-slave-9tkz krb5kdc[30620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836748, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:08.844941 30633 init.cc:377] Logged in from keytab as kudu/127.7.194.254@KRBTEST.COM (short username kudu)
I20260619 02:39:08.845983 30633 webserver.cc:460] Webserver started at http://127.7.194.254:36927/ using document root <none> and password file <none>
I20260619 02:39:08.846560 30633 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:08.846654 30633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:08.846897 30633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:08.848969 30633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/data/instance:
uuid: "3536926221674fe5a461566b99178a6c"
format_stamp: "Formatted at 2026-06-19 02:39:08 on dist-test-slave-9tkz"
server_key: "0fb595c73c3e5fd3729378d7f752368a"
server_key_iv: "4d7799ba59cfc95401dff6ceb09e78ea"
server_key_version: "encryptionkey@0"
I20260619 02:39:08.849529 30633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/wal/instance:
uuid: "3536926221674fe5a461566b99178a6c"
format_stamp: "Formatted at 2026-06-19 02:39:08 on dist-test-slave-9tkz"
server_key: "0fb595c73c3e5fd3729378d7f752368a"
server_key_iv: "4d7799ba59cfc95401dff6ceb09e78ea"
server_key_version: "encryptionkey@0"
I20260619 02:39:08.853283 30633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:39:08.855772 30648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:08.856953 30633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:39:08.857101 30633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/wal
uuid: "3536926221674fe5a461566b99178a6c"
format_stamp: "Formatted at 2026-06-19 02:39:08 on dist-test-slave-9tkz"
server_key: "0fb595c73c3e5fd3729378d7f752368a"
server_key_iv: "4d7799ba59cfc95401dff6ceb09e78ea"
server_key_version: "encryptionkey@0"
I20260619 02:39:08.857214 30633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:08.873471 30633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:08.876335 30633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:08.876541 30633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:08.883695 30633 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.254:38217
I20260619 02:39:08.883986 30700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.254:38217 every 8 connection(s)
I20260619 02:39:08.884682 30633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/data/info.pb
I20260619 02:39:08.887439 30701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:08.889632 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 30633
I20260619 02:39:08.889778 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-0/wal/instance
I20260619 02:39:08.890115 7947 external_mini_cluster.cc:1428] Setting key 259fbfed161475f958b952fddd781ca0
I20260619 02:39:08.892125 30701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.194.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.253@KRBTEST.COM" created.
Jun 19 02:39:08 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836748, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:39:08.907632 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:08.894414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.7.194.254:38217)
0619 02:39:08.894941 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:08.894955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:08.895007 (+ 52us) server_negotiation.cc:407] Connection header received
0619 02:39:08.896064 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:08.896072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:08.896349 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:08.896563 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:08.897335 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:08.898850 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:08.899788 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:08.900261 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:08.900405 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:08.902705 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:08.902717 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:08.902728 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:08.902749 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:08.904483 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:08.904933 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:08.904939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:08.904940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:08.905043 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:08.905359 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:08.905365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:08.905366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:08.905824 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:08.905976 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:08.906327 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 02:39:08.906612 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":396,"spinlock_wait_cycles":13696,"thread_start_us":131,"threads_started":1}
I20260619 02:39:08.907632 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:08.894413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38217 (local address 127.0.0.1:44172)
0619 02:39:08.894871 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:08.894922 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:08.895738 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:08.896561 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:08.896565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:08.896889 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:08.897147 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:08.897178 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:08.898962 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:08.898964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:08.899678 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:08.899687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:08.900411 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:08.900414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:08.900503 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:08.901096 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:08.901112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:08.902558 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:08.904586 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:08.904595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:08.904605 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:08.904827 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:08.905139 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:08.905141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:08.905143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:08.905270 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:08.905935 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:08.905941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:08.906093 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 02:39:08.906262 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 02:39:08.906659 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":111,"thread_start_us":137,"threads_started":1}
I20260619 02:39:08.909911 30701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:08.910737 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:08.910466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33593 (local address 127.0.0.1:38956)
0619 02:39:08.910600 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:08.910674 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 02:39:08.910928 30652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
I20260619 02:39:08.911839 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:08.911623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33593 (local address 127.0.0.1:38962)
0619 02:39:08.911723 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:08.911786 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 02:39:08.912011 30701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.253:33593: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab.
Entry for principal kudu/127.7.194.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab.
I20260619 02:39:08.915925 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.253:33593
--webserver_interface=127.7.194.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab
--principal=kudu/127.7.194.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38217,127.7.194.253:33593,127.7.194.252:33101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:39:08.961239 30701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } attempt: 1
I20260619 02:39:08.962146 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:08.961814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33593 (local address 127.0.0.1:38966)
0619 02:39:08.962003 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:08.962068 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 02:39:08.962869 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:08.962651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33593 (local address 127.0.0.1:38978)
0619 02:39:08.962770 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:08.962817 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 02:39:08.962994 30701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.253:33593: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
W20260619 02:39:09.026638 30708 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:39:09.026995 30708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:09.027072 30708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:09.027149 30708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:39:09.031138 30708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:39:09.031225 30708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260619 02:39:09.031150 30701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } attempt: 2
W20260619 02:39:09.031266 30708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:39:09.031298 30708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:09.031334 30708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:09.032023 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.031685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33593 (local address 127.0.0.1:38992)
0619 02:39:09.031869 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.031941 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 02:39:09.032732 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.032522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33593 (local address 127.0.0.1:39002)
0619 02:39:09.032630 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.032679 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 02:39:09.032863 30701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.253:33593: Network error: Client connection negotiation failed: client connection to 127.7.194.253:33593: connect: Connection refused (error 111)
I20260619 02:39:09.036940 30708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38217,127.7.194.253:33593,127.7.194.252:33101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.253.keytab
--principal=kudu/127.7.194.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.253:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.194.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:09.038009 30708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:09.038972 30708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:39:09.047348 30713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:09.047348 30714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:09.047433 30708 server_base.cc:1034] running on GCE node
W20260619 02:39:09.047384 30716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:09.048168 30708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:09.049290 30708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:09.050451 30708 hybrid_clock.cc:648] HybridClock initialized: now 1781836749050410 us; error 52 us; skew 500 ppm
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:09.053182 30708 init.cc:377] Logged in from keytab as kudu/127.7.194.253@KRBTEST.COM (short username kudu)
I20260619 02:39:09.054155 30708 webserver.cc:460] Webserver started at http://127.7.194.253:38339/ using document root <none> and password file <none>
I20260619 02:39:09.054705 30708 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:09.054793 30708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:09.055035 30708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:09.057176 30708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/data/instance:
uuid: "b55b68d7a7be41be888414c91b672aad"
format_stamp: "Formatted at 2026-06-19 02:39:09 on dist-test-slave-9tkz"
server_key: "3b373595f7ec09bbb45555d3e5fe6f66"
server_key_iv: "97ef140b7a65eb778121b678d2231d0e"
server_key_version: "encryptionkey@0"
I20260619 02:39:09.057818 30708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/wal/instance:
uuid: "b55b68d7a7be41be888414c91b672aad"
format_stamp: "Formatted at 2026-06-19 02:39:09 on dist-test-slave-9tkz"
server_key: "3b373595f7ec09bbb45555d3e5fe6f66"
server_key_iv: "97ef140b7a65eb778121b678d2231d0e"
server_key_version: "encryptionkey@0"
I20260619 02:39:09.061658 30708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:39:09.064003 30723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:09.065232 30708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:39:09.065373 30708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/wal
uuid: "b55b68d7a7be41be888414c91b672aad"
format_stamp: "Formatted at 2026-06-19 02:39:09 on dist-test-slave-9tkz"
server_key: "3b373595f7ec09bbb45555d3e5fe6f66"
server_key_iv: "97ef140b7a65eb778121b678d2231d0e"
server_key_version: "encryptionkey@0"
I20260619 02:39:09.065490 30708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:09.091646 30708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:09.095081 30708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:09.095295 30708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:09.102212 30708 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.253:33593
I20260619 02:39:09.102520 30775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.253:33593 every 8 connection(s)
I20260619 02:39:09.103202 30708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/data/info.pb
I20260619 02:39:09.106074 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:09.109808 30776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:09.112351 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 30708
I20260619 02:39:09.112495 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-1/wal/instance
I20260619 02:39:09.112835 7947 external_mini_cluster.cc:1428] Setting key 111d1fbfddc623919e7f7ff9cfd4454c
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.194.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.194.252@KRBTEST.COM" created.
I20260619 02:39:09.124030 30701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } attempt: 3
I20260619 02:39:09.125337 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.111653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.7.194.254:38217)
0619 02:39:09.111827 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.111831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.112743 (+ 912us) server_negotiation.cc:407] Connection header received
0619 02:39:09.114163 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.114166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.114218 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.114301 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.115564 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.116099 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.117334 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.117637 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.117751 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.120378 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.120381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.120383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.120408 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.122576 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.123218 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.123221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.123222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.123310 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.123713 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.123715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.123716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.123894 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.123970 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.124964 (+ 994us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.125111 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:39:09.126317 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.112043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38217 (local address 127.0.0.1:44186)
0619 02:39:09.112556 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.112616 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:09.113930 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:09.114510 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:09.114523 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:09.114929 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:09.115411 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.115425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.116229 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.116239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.117210 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.117219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.117802 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.117806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.117981 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.118676 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:09.118693 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:09.120185 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:09.122729 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.122737 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.122753 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.123102 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.123412 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.123418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.123423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.123621 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.124016 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:09.124025 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:09.124394 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 02:39:09.124880 (+ 486us) client_negotiation.cc:240] Negotiation successful
0619 02:39:09.125227 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260619 02:39:09.127794 30776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836748, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
I20260619 02:39:09.134832 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.124492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33593 (local address 127.0.0.1:39012)
0619 02:39:09.124629 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.124644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:09.124709 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:09.125367 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:09.125370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:09.125532 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:09.125684 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.125691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.127085 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.127091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.127888 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.127898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.128693 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.128698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.128809 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.129263 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:09.129273 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:09.130870 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:09.133159 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.133162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.133164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.133385 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.133736 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.133740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.133743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.133841 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.134417 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:09.134419 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:09.134500 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 02:39:09.134579 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:39:09.134686 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:39:09.135079 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.124580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.7.194.253:33593)
0619 02:39:09.124918 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.124931 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.124962 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:39:09.125000 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.125005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.125207 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.125835 (+ 628us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.125936 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.126973 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.128021 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.128552 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.129007 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.131028 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.131038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.131052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.131079 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.133056 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.133499 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.133507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.133511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.133630 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.133949 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.133952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.133954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.134289 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.134440 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.134713 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.134873 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":117,"threads_started":1}
I20260619 02:39:09.137311 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.128343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33593 (local address 127.0.0.1:39028)
0619 02:39:09.128479 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.128497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:09.128579 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:09.129019 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:09.129023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:09.129195 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:09.129347 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.129357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.130269 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.130272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.130841 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.130850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.131302 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.131305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.131422 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.131822 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:09.131833 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:09.133146 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:09.135912 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.135915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.135916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.136102 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.136373 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.136378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.136380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.136454 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.136802 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:09.136804 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:09.136881 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 02:39:09.137009 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 02:39:09.137127 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":18}
I20260619 02:39:09.137277 30701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:09.137705 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.128486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.7.194.253:33593)
0619 02:39:09.128791 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.128799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.128810 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:39:09.128841 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.128844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.128890 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.128983 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.129479 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.130153 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.130947 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.131202 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.131305 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.133262 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.133264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.133266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.133283 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.135804 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.136202 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.136208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.136210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.136284 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.136545 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.136548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.136550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.136705 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.136814 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.137398 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.137526 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260619 02:39:09.137970 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.137765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45550)
0619 02:39:09.137861 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.137916 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 02:39:09.138526 30776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:09.138725 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.138488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45566)
0619 02:39:09.138623 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.138669 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 02:39:09.138859 30701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:33101: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
I20260619 02:39:09.139192 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.138987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45578)
0619 02:39:09.139076 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.139144 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 02:39:09.139405 30726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
I20260619 02:39:09.140256 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.140041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45586)
0619 02:39:09.140128 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.140186 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.194.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab.
Entry for principal kudu/127.7.194.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab.
W20260619 02:39:09.140507 30776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:33101: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
I20260619 02:39:09.141168 7947 external_mini_cluster.cc:1326] Running /tmp/dist-test-task73xKWu/build/debug/bin/kudu
/tmp/dist-test-task73xKWu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.252:33101
--webserver_interface=127.7.194.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab
--principal=kudu/127.7.194.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.194.212:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.194.254:38217,127.7.194.253:33593,127.7.194.252:33101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:39:09.170061 30701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } attempt: 1
I20260619 02:39:09.171021 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.170723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45588)
0619 02:39:09.170874 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.170939 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 02:39:09.171729 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.171519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45598)
0619 02:39:09.171623 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.171675 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 02:39:09.171865 30701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:33101: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
I20260619 02:39:09.189738 30776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } attempt: 1
I20260619 02:39:09.190659 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.190359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45612)
0619 02:39:09.190512 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.190579 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 02:39:09.191416 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.191188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45614)
0619 02:39:09.191293 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.191359 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 02:39:09.191548 30776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:33101: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
I20260619 02:39:09.247025 30701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } attempt: 2
I20260619 02:39:09.247884 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.247520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45624)
0619 02:39:09.247718 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.247789 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 02:39:09.248625 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.248404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45632)
0619 02:39:09.248510 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.248563 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:39:09.248759 30701 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:33101: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
W20260619 02:39:09.253258 30784 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:39:09.253610 30784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:39:09.253662 30784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:39:09.253736 30784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:39:09.258412 30784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:39:09.258472 30784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:39:09.258512 30784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:39:09.258553 30784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:39:09.258599 30784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:39:09.259697 30776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } attempt: 2
I20260619 02:39:09.260481 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.260174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45636)
0619 02:39:09.260337 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.260401 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:39:09.261312 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.261080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45652)
0619 02:39:09.261201 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.261249 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 02:39:09.261454 30776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.194.252:33101: Network error: Client connection negotiation failed: client connection to 127.7.194.252:33101: connect: Connection refused (error 111)
I20260619 02:39:09.264684 30784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.194.212:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.194.254:38217,127.7.194.253:33593,127.7.194.252:33101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/krb5kdc/kudu_127.7.194.252.keytab
--principal=kudu/127.7.194.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.194.252:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.194.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:39:09.265877 30784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:39:09.267023 30784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:39:09.275628 30792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:09.275686 30789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:39:09.275640 30790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:39:09.275951 30784 server_base.cc:1034] running on GCE node
I20260619 02:39:09.276536 30784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:39:09.277693 30784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:39:09.278851 30784 hybrid_clock.cc:648] HybridClock initialized: now 1781836749278817 us; error 45 us; skew 500 ppm
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:39:09.281841 30784 init.cc:377] Logged in from keytab as kudu/127.7.194.252@KRBTEST.COM (short username kudu)
I20260619 02:39:09.282753 30784 webserver.cc:460] Webserver started at http://127.7.194.252:39383/ using document root <none> and password file <none>
I20260619 02:39:09.283306 30784 fs_manager.cc:359] Metadata directory not provided
I20260619 02:39:09.283407 30784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:39:09.283679 30784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:39:09.285774 30784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/data/instance:
uuid: "3e6ab428e28f43d0a4917cbebb348779"
format_stamp: "Formatted at 2026-06-19 02:39:09 on dist-test-slave-9tkz"
server_key: "98a655a7f4dd34867a69714b4845bc0b"
server_key_iv: "709c439336f5d48a8834e90dc8514411"
server_key_version: "encryptionkey@0"
I20260619 02:39:09.286355 30784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/wal/instance:
uuid: "3e6ab428e28f43d0a4917cbebb348779"
format_stamp: "Formatted at 2026-06-19 02:39:09 on dist-test-slave-9tkz"
server_key: "98a655a7f4dd34867a69714b4845bc0b"
server_key_iv: "709c439336f5d48a8834e90dc8514411"
server_key_version: "encryptionkey@0"
I20260619 02:39:09.290107 30784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:39:09.292542 30799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:39:09.293823 30784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:39:09.293958 30784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/data,/tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/wal
uuid: "3e6ab428e28f43d0a4917cbebb348779"
format_stamp: "Formatted at 2026-06-19 02:39:09 on dist-test-slave-9tkz"
server_key: "98a655a7f4dd34867a69714b4845bc0b"
server_key_iv: "709c439336f5d48a8834e90dc8514411"
server_key_version: "encryptionkey@0"
I20260619 02:39:09.294068 30784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:39:09.307490 30784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:39:09.311813 30784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:39:09.312054 30784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:39:09.319988 30784 rpc_server.cc:307] RPC server started. Bound to: 127.7.194.252:33101
I20260619 02:39:09.320215 30851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.194.252:33101 every 8 connection(s)
I20260619 02:39:09.321056 30784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/data/info.pb
I20260619 02:39:09.323890 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:39:09.327431 7947 external_mini_cluster.cc:1388] Started /tmp/dist-test-task73xKWu/build/debug/bin/kudu as pid 30784
I20260619 02:39:09.327612 7947 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task73xKWu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781836464269074-7947-0/minicluster-data/master-2/wal/instance
I20260619 02:39:09.327935 7947 external_mini_cluster.cc:1428] Setting key b28c7f8ddef71eac50435b61626f9621
I20260619 02:39:09.327663 30852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
I20260619 02:39:09.340451 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.329715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.7.194.254:38217)
0619 02:39:09.329874 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.329879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.330182 (+ 303us) server_negotiation.cc:407] Connection header received
0619 02:39:09.330911 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.330913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.330960 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.331032 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.331979 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.332469 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.333598 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.333910 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.333988 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.336608 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.336611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.336613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.336636 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.338301 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.338875 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.338879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.338881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.338961 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.339308 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.339312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.339314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.339511 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.339615 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.340128 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.340248 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:39:09.341287 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.329632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.254:38217 (local address 127.0.0.1:44190)
0619 02:39:09.330041 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.330086 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:09.330690 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:09.331106 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:09.331111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:09.331559 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:09.331834 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.331848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.332573 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.332577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.333429 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.333436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.334041 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.334049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.334220 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.334853 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:09.334868 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:09.336461 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:09.338432 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.338442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.338452 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.338764 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.339064 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.339067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.339069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.339206 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.339623 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:09.339633 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:09.339908 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 02:39:09.340075 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 02:39:09.340387 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":23,"spinlock_wait_cycles":768,"thread_start_us":167,"threads_started":1}
I20260619 02:39:09.341292 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.329266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.7.194.252:33101)
0619 02:39:09.329748 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.329767 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.329832 (+ 65us) server_negotiation.cc:407] Connection header received
0619 02:39:09.330690 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.330713 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.330952 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.331138 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.331848 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.332841 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.333818 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.334266 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.334372 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.337007 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.337013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.337022 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.337061 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.338726 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.339161 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.339168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.339171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.339249 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.339530 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.339533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.339534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.339897 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.340101 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.340284 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.340436 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":332,"thread_start_us":120,"threads_started":1}
I20260619 02:39:09.342588 30852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:09.347908 30701 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } attempt: 3
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.254@KRBTEST.COM
I20260619 02:39:09.352613 30776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } attempt: 3
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836748, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.254@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
I20260619 02:39:09.353641 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.343224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.7.194.253:33593)
0619 02:39:09.343385 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.343392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.343409 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:39:09.343572 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.343576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.343634 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.343729 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.344185 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.344923 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.345836 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.346191 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.346292 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.349166 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.349170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.349173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.349202 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.350985 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.351494 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.351500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.351501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.351588 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.352504 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.352507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.352509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.352731 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.352813 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.353174 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.353298 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:39:09.354175 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.343087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.253:33593 (local address 127.0.0.1:39036)
0619 02:39:09.343362 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.343374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:09.343462 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:09.343741 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:09.343743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:09.343929 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:09.344063 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.344071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.345023 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.345026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.345734 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.345742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.346304 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.346308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.346405 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.346961 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:09.346977 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:09.348985 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:09.351127 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.351130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.351132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.351404 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.351701 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.351704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.351706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.351801 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.352860 (+ 1059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:09.352863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:09.352958 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 02:39:09.353831 (+ 873us) client_negotiation.cc:240] Negotiation successful
0619 02:39:09.353973 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"spinlock_wait_cycles":9472}
I20260619 02:39:09.354610 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.343557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.7.194.254:38217)
0619 02:39:09.343674 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.343678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.343792 (+ 114us) server_negotiation.cc:407] Connection header received
0619 02:39:09.343880 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.343883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.343928 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.344012 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.344695 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.345292 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.346263 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.346591 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.346693 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.349193 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.349196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.349199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.349227 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.351593 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.352055 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.352062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.352064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.352149 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.352388 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.352395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.352398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.352651 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.352878 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.354063 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.354191 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:39:09.355458 30852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:39:09.357107 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.348307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45674)
0619 02:39:09.348433 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.348449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:09.348521 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:09.348867 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:09.348870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:09.349042 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:09.349159 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.349166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.350198 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.350203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.350710 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.350718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.351341 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.351346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.351444 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.351913 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:09.351926 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:09.353360 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:09.355407 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.355410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.355412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.355639 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.356109 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.356112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.356114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.356204 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.356657 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:09.356660 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:09.356752 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 02:39:09.356852 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 02:39:09.356968 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:39:09.357864 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.348557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.7.194.252:33101)
0619 02:39:09.348662 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.348666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.348677 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:39:09.348716 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.348718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.348750 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.348848 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.349273 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.350088 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.350847 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.351217 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.351339 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.353506 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.353510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.353513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.353536 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.355284 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.355768 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.355771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.355772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.355863 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.356324 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.356328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.356330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.356541 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.356633 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.357600 (+ 967us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.357722 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.253@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.194.252@KRBTEST.COM for kudu/127.7.194.252@KRBTEST.COM
I20260619 02:39:09.362118 30701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c: Bootstrap starting.
Jun 19 02:39:09 dist-test-slave-9tkz krb5kdc[30620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.194.253@KRBTEST.COM
I20260619 02:39:09.365475 30701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:09.365880 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.353038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45680)
0619 02:39:09.353244 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.353259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:09.353348 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:09.354843 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:09.354846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:09.355014 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:09.355243 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.355256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.357377 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.357383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.358245 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.358254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.359014 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.359018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.359098 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.359561 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:09.359575 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:09.361160 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:09.363459 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.363465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.363470 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.364026 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.364364 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.364369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.364371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.364469 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.365360 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:09.365363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:09.365448 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 02:39:09.365530 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:39:09.365689 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":22}
I20260619 02:39:09.366015 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.356005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.194.252:33101 (local address 127.0.0.1:45684)
0619 02:39:09.356091 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 02:39:09.356107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:39:09.356191 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:39:09.357008 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:39:09.357013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:39:09.357186 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:39:09.357267 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.357273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.358411 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.358414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.359158 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:39:09.359168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.360073 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.360079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:39:09.360175 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.360595 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:39:09.360607 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:39:09.362139 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:39:09.364566 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.364570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.364572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.364806 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.365103 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:39:09.365106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:39:09.365108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:39:09.365172 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:39:09.365525 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:39:09.365527 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:39:09.365652 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 02:39:09.365732 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:39:09.365860 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 02:39:09.366608 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.356184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.7.194.252:33101)
0619 02:39:09.356772 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.356783 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.356809 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:39:09.356840 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.356843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.356884 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.357374 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.357512 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.358297 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.359560 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.359902 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.360363 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.362275 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.362281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.362286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.362317 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.364463 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.364911 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.364917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.364922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.365015 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.365263 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.365265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.365267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.365433 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.366278 (+ 845us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.366353 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.366452 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":102,"threads_started":1}
I20260619 02:39:09.366662 30701 log.cc:824] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:09.367034 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.356120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.7.194.253:33593)
0619 02:39:09.356337 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.356341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.356357 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:39:09.356506 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.356509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.356551 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.356601 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.357465 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.358267 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.359271 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.359610 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.359683 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.363082 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.363086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.363088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.363112 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.364886 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.365342 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.365348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.365353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.365438 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.365944 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.365947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.365951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.366192 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.366376 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.366787 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.366899 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":146}
I20260619 02:39:09.367348 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.353571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.7.194.252:33101)
0619 02:39:09.354577 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.354586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.354610 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:39:09.354644 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.354647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.354694 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.355382 (+ 688us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.355492 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.357235 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.358534 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.358890 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.359317 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.361310 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.361313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.361316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.361340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.363310 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.364137 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.364155 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.364157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.364244 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.365029 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.365034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.365039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.365260 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.366894 (+ 1634us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.366961 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.367094 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":125,"threads_started":1}
I20260619 02:39:09.369828 30852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779: Bootstrap starting.
I20260619 02:39:09.370357 30776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad: Bootstrap starting.
I20260619 02:39:09.370985 7947 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:39:09.372319 30852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:09.372525 30776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad: Neither blocks nor log segments found. Creating new log.
I20260619 02:39:09.373005 30852 log.cc:824] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:09.373327 30776 log.cc:824] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad: Log is configured to *not* fsync() on all Append() calls
I20260619 02:39:09.372177 30675 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3536926221674fe5a461566b99178a6c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44198
I20260619 02:39:09.375145 30852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779: No bootstrap required, opened a new log
I20260619 02:39:09.375314 30776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad: No bootstrap required, opened a new log
I20260619 02:39:09.375314 30701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c: No bootstrap required, opened a new log
I20260619 02:39:09.378264 30701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } }
I20260619 02:39:09.378561 30701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3536926221674fe5a461566b99178a6c, State: Initialized, Role: FOLLOWER
I20260619 02:39:09.378917 30701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } }
I20260619 02:39:09.379185 30776 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } }
I20260619 02:39:09.379380 30776 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55b68d7a7be41be888414c91b672aad, State: Initialized, Role: FOLLOWER
I20260619 02:39:09.379675 30776 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } }
I20260619 02:39:09.379717 30852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } }
I20260619 02:39:09.379926 30852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6ab428e28f43d0a4917cbebb348779, State: Initialized, Role: FOLLOWER
I20260619 02:39:09.380211 30852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } }
I20260619 02:39:09.380332 30701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:09.380492 30675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:39:09.380604 30675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:09.380954 30776 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:09.381379 30852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:39:09.381918 30675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } }
I20260619 02:39:09.385725 30863 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.385879 30863 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:39:09.386536 30885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:39:09.386627 30885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 02:39:09.387555 30896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:39:09.387797 30896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:39:09.387971 30862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.388065 30862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:09.389165 30784 master_runner.cc:232] Existing masters have differing terms: 1 vs 0
I20260619 02:39:09.389263 30784 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:39:09.390066 30675 leader_election.cc:288] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [CANDIDATE]: Term 1 election: Requested vote from peers b55b68d7a7be41be888414c91b672aad (127.7.194.253:33593), 3e6ab428e28f43d0a4917cbebb348779 (127.7.194.252:33101)
I20260619 02:39:09.390586 30826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3536926221674fe5a461566b99178a6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3e6ab428e28f43d0a4917cbebb348779"
I20260619 02:39:09.390798 30826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:09.391774 30826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3536926221674fe5a461566b99178a6c in term 1.
I20260619 02:39:09.392139 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3536926221674fe5a461566b99178a6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b55b68d7a7be41be888414c91b672aad"
I20260619 02:39:09.392367 30750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:39:09.393321 30750 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3536926221674fe5a461566b99178a6c in term 1.
I20260619 02:39:09.392163 30651 leader_election.cc:302] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3536926221674fe5a461566b99178a6c, 3e6ab428e28f43d0a4917cbebb348779; no voters:
I20260619 02:39:09.394363 30862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:39:09.395056 30633 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260619 02:39:09.395179 30633 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:39:09.395248 30862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [term 1 LEADER]: Becoming Leader. State: Replica: 3536926221674fe5a461566b99178a6c, State: Running, Role: LEADER
I20260619 02:39:09.395560 30862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } }
W20260619 02:39:09.396965 30897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:39:09.397047 30897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:39:09.397501 30862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3536926221674fe5a461566b99178a6c. Latest consensus state: current_term: 1 leader_uuid: "3536926221674fe5a461566b99178a6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.397622 30862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:09.397835 30708 master_runner.cc:263] No leader master found from master b55b68d7a7be41be888414c91b672aad
I20260619 02:39:09.397897 30708 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:39:09.398604 30908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:39:09.399555 30908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:39:09.407171 30864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.407333 30864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:09.411113 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.396012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.7.194.252:33101)
0619 02:39:09.396157 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.396162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.398125 (+ 1963us) server_negotiation.cc:407] Connection header received
0619 02:39:09.398275 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.398278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.398326 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.398410 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.402486 (+ 4076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.403335 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.404589 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.405000 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.405095 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.407486 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.407490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.407493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.407518 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.409243 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.409812 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.409816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.409818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.409899 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.410198 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.410201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.410203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.410411 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.410522 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.410842 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.410954 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:39:09.422220 30826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [term 1 FOLLOWER]: Refusing update from remote peer 3536926221674fe5a461566b99178a6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:39:09.423041 30750 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [term 1 FOLLOWER]: Refusing update from remote peer 3536926221674fe5a461566b99178a6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:39:09.423493 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.405949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.7.194.253:33593)
0619 02:39:09.406070 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.406074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.406733 (+ 659us) server_negotiation.cc:407] Connection header received
0619 02:39:09.406907 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.406910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.406960 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.407011 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.407847 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.408633 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.417669 (+ 9036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.417973 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.418053 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.419938 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.419942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.419944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.419975 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.421695 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.422209 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.422212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.422214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.422297 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.422574 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.422577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.422579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.422779 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.422870 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.423226 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.423338 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:39:09.423735 30902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:39:09.423920 30862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:39:09.427799 30863 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3536926221674fe5a461566b99178a6c. Latest consensus state: current_term: 1 leader_uuid: "3536926221674fe5a461566b99178a6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.427929 30863 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:09.428071 30864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3536926221674fe5a461566b99178a6c. Latest consensus state: current_term: 1 leader_uuid: "3536926221674fe5a461566b99178a6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.428195 30864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:09.428900 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:09.407461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.7.194.254:38217)
0619 02:39:09.407580 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.407585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.412664 (+ 5079us) server_negotiation.cc:407] Connection header received
0619 02:39:09.412833 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.412837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.412884 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.412931 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.413812 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.414576 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.415697 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:09.416017 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:09.416082 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:09.417768 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:39:09.417773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:39:09.417775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:39:09.417802 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:39:09.419451 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.419963 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.419967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.419969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.420049 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:39:09.420352 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:39:09.420356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:39:09.420358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:39:09.420558 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:39:09.420663 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:09.428606 (+ 7943us) server_negotiation.cc:299] Negotiation successful
0619 02:39:09.428739 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:39:09.433707 30902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3536926221674fe5a461566b99178a6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.433856 30902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:09.435307 30864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3536926221674fe5a461566b99178a6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.435421 30864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:09.437019 30862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3536926221674fe5a461566b99178a6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.437148 30862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c [sys.catalog]: This master's current role is: LEADER
I20260619 02:39:09.437496 30912 mvcc.cc:205] Tried to move back new op lower bound from 7298403325622886400 to 7298403325528457216. Current Snapshot: MvccSnapshot[applied={T|T < 7298403325622886400}]
I20260619 02:39:09.438061 30863 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3536926221674fe5a461566b99178a6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3536926221674fe5a461566b99178a6c" member_type: VOTER last_known_addr { host: "127.7.194.254" port: 38217 } } peers { permanent_uuid: "b55b68d7a7be41be888414c91b672aad" member_type: VOTER last_known_addr { host: "127.7.194.253" port: 33593 } } peers { permanent_uuid: "3e6ab428e28f43d0a4917cbebb348779" member_type: VOTER last_known_addr { host: "127.7.194.252" port: 33101 } } }
I20260619 02:39:09.438195 30863 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:39:09.441706 30913 mvcc.cc:205] Tried to move back new op lower bound from 7298403325622886400 to 7298403325528457216. Current Snapshot: MvccSnapshot[applied={T|T < 7298403325622886400}]
I20260619 02:39:09.444561 30908 catalog_manager.cc:1347] Generated new cluster ID: c8db8152c89d408fb1b549a49d6fd5d2
I20260619 02:39:09.444635 30908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:39:09.468611 30908 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:39:09.472304 30908 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:39:09.487175 30908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3536926221674fe5a461566b99178a6c: Generated new TSK 0
I20260619 02:39:09.488121 30908 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:39:09.521903 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.519319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.7.194.253:33593)
0619 02:39:09.519446 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.519450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.520702 (+ 1252us) server_negotiation.cc:407] Connection header received
0619 02:39:09.520943 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.520947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.520998 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.521051 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.521755 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39066: BlockingRecv error: recv got EOF from 127.0.0.1:39066 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 02:39:09.523442 30857 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.520085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.7.194.252:33101)
0619 02:39:09.520201 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.520205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.522304 (+ 2099us) server_negotiation.cc:407] Connection header received
0619 02:39:09.522686 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.522690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.522741 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.522792 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.523327 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45696: BlockingRecv error: recv got EOF from 127.0.0.1:45696 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 02:39:09.525341 30705 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.523901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.7.194.254:38217)
0619 02:39:09.524007 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.524011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.524477 (+ 466us) server_negotiation.cc:407] Connection header received
0619 02:39:09.524638 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.524641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.524693 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.524742 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.525217 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44216: BlockingRecv error: recv got EOF from 127.0.0.1:44216 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 02:39:09.529757 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.528245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.7.194.253:33593)
0619 02:39:09.528349 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.528353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.528768 (+ 415us) server_negotiation.cc:407] Connection header received
0619 02:39:09.528980 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.528983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.529059 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.529109 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.529655 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39070: BlockingRecv error: recv got EOF from 127.0.0.1:39070 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 02:39:09.534919 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.533266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.7.194.253:33593)
0619 02:39:09.533365 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.533369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.533979 (+ 610us) server_negotiation.cc:407] Connection header received
0619 02:39:09.534179 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.534183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.534238 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.534289 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.534813 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39080: BlockingRecv error: recv got EOF from 127.0.0.1:39080 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 02:39:09.540612 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.539016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.7.194.253:33593)
0619 02:39:09.539117 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.539121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.539642 (+ 521us) server_negotiation.cc:407] Connection header received
0619 02:39:09.539832 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.539838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.539887 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.539941 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.540491 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39094: BlockingRecv error: recv got EOF from 127.0.0.1:39094 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 02:39:09.548550 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.546859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.7.194.253:33593)
0619 02:39:09.547010 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.547015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.547594 (+ 579us) server_negotiation.cc:407] Connection header received
0619 02:39:09.547798 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.547801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.547855 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.547905 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.548436 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39102: BlockingRecv error: recv got EOF from 127.0.0.1:39102 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 02:39:09.561031 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.559023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.7.194.253:33593)
0619 02:39:09.559169 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.559174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.559792 (+ 618us) server_negotiation.cc:407] Connection header received
0619 02:39:09.560018 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.560021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.560081 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.560138 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.560907 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39104: BlockingRecv error: recv got EOF from 127.0.0.1:39104 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 02:39:09.582427 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.579846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.7.194.253:33593)
0619 02:39:09.580008 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.580013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.580803 (+ 790us) server_negotiation.cc:407] Connection header received
0619 02:39:09.581012 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.581016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.581076 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.581171 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.582301 (+ 1130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39116: BlockingRecv error: recv got EOF from 127.0.0.1:39116 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 02:39:09.618223 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.616957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.7.194.253:33593)
0619 02:39:09.617109 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.617113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.617255 (+ 142us) server_negotiation.cc:407] Connection header received
0619 02:39:09.617437 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.617440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.617497 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.617629 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.618100 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39126: BlockingRecv error: recv got EOF from 127.0.0.1:39126 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 02:39:09.685665 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.684426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.7.194.253:33593)
0619 02:39:09.684600 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.684604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.684678 (+ 74us) server_negotiation.cc:407] Connection header received
0619 02:39:09.684804 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.684807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.684859 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.684935 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.685371 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39132: BlockingRecv error: recv got EOF from 127.0.0.1:39132 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 02:39:09.817050 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:09.815788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.7.194.253:33593)
0619 02:39:09.815938 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:09.815943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:09.816134 (+ 191us) server_negotiation.cc:407] Connection header received
0619 02:39:09.816310 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:09.816315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:09.816375 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:09.816468 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:09.816943 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39136: BlockingRecv error: recv got EOF from 127.0.0.1:39136 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 02:39:10.076598 30782 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:39:10.075380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.7.194.253:33593)
0619 02:39:10.075568 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:10.075573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:10.075677 (+ 104us) server_negotiation.cc:407] Connection header received
0619 02:39:10.075884 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:10.075887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:10.075937 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:10.076026 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:39:10.076476 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39142: BlockingRecv error: recv got EOF from 127.0.0.1:39142 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:39:10.389030 30896 catalog_manager.cc:1259] Loaded cluster ID: c8db8152c89d408fb1b549a49d6fd5d2
I20260619 02:39:10.389117 30896 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad: loading cluster ID for follower catalog manager: success
I20260619 02:39:10.390763 30896 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad: acquiring CA information for follower catalog manager: success
I20260619 02:39:10.391885 30896 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b55b68d7a7be41be888414c91b672aad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:39:10.398274 30897 catalog_manager.cc:1259] Loaded cluster ID: c8db8152c89d408fb1b549a49d6fd5d2
I20260619 02:39:10.398370 30897 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779: loading cluster ID for follower catalog manager: success
I20260619 02:39:10.399938 30897 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779: acquiring CA information for follower catalog manager: success
I20260619 02:39:10.401072 30897 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e6ab428e28f43d0a4917cbebb348779: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:39:10.594811 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:10.591001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.7.194.253:33593)
0619 02:39:10.591258 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:10.591262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:10.591346 (+ 84us) server_negotiation.cc:407] Connection header received
0619 02:39:10.591481 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:10.591484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:10.591610 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:10.591712 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:10.591947 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.592480 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.593298 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.593528 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.593673 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:10.593962 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:10.594404 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:10.594496 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:10.594550 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 02:39:10.594598 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
I20260619 02:39:10.601521 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:10.597965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.7.194.252:33101)
0619 02:39:10.598221 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:10.598224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:10.598238 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:39:10.598350 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:10.598352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:10.598471 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:10.598565 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:10.598840 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.599349 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.600128 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.600350 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.600429 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:10.600671 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:10.601104 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:10.601198 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:10.601254 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 02:39:10.601302 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260619 02:39:10.608606 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:10.604528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.7.194.253:33593)
0619 02:39:10.604723 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:10.604728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:10.605116 (+ 388us) server_negotiation.cc:407] Connection header received
0619 02:39:10.605243 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:10.605249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:10.605308 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:10.605396 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:10.605671 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.606253 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.607347 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.607704 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.607803 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:10.608063 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:10.608167 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:10.608234 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:10.608383 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 02:39:10.608444 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 02:39:10.608744 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:10.604734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.7.194.252:33101)
0619 02:39:10.604842 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:10.604848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:10.605177 (+ 329us) server_negotiation.cc:407] Connection header received
0619 02:39:10.605298 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:10.605301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:10.605344 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:10.605412 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:10.605866 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.606608 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.607537 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.607845 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.607940 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:10.608225 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:10.608309 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:10.608375 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:10.608563 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 02:39:10.608615 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:39:10.609866 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:39:10.604843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.7.194.254:38217)
0619 02:39:10.605105 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 02:39:10.605110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:39:10.605124 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:39:10.605236 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:39:10.605239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:39:10.605409 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:39:10.605523 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:39:10.605780 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.606612 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.607665 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:39:10.608031 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:39:10.608148 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:39:10.608474 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:39:10.609274 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:39:10.609367 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:39:10.609674 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 02:39:10.609741 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260619 02:39:10.612936 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 30633
I20260619 02:39:10.619252 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 30708
I20260619 02:39:10.625327 7947 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task73xKWu/build/debug/bin/kudu with pid 30784
2026-06-19T02:39:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3081 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6187 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286349 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST